Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Jun 2012 - 30 Jun 2012 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 72,349,370 page requests (mime type text/html only!) per day are considered crawler requests, out of 489,483,170 external requests, which is 14.8%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cloudcrawling)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
 code.google.com/p/crawler4j/text/..crawler4j (url)
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~senchaiosrc)
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
 code.google.com/appengineimage/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy)
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
 code.google.com/appenginetext/..Mozilla/5.0 AppEngine-Google; (url; appid: s~app3123ak)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~wikigraph2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hexowiki-bd-citycell)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wagagate)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3)
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 114proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda)
 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crowdsurfer100)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2)
 code.google.com/p/rondaapplication/jsonRonda - url
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bie99miracle)
 www.google.com/bot.htmlNONE/wikipedia- Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10)
 code.google.com/appenginetext/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki1)
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web4proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: seiyukyouen)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps002)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nhsportal)
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp8)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: islom-proxy)
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: argim-free)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools6)
 code.google.com/p/rondatext/..Ronda - url
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: issamwap1)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sl-santabanta)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq0)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn03)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps001)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kttpxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openeyeproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp6)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ageryder)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pramod-ajgaonkar-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-5)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog)
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-4)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kha1989led)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools9)
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 developers.facebook.comimage/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
 developers.facebook.com-facebookplatform/1.0 (url)
 developers.facebook.comtext/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url)
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html))
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy6)
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: misc-tools)
 www.baidu.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
 www.baidu.com/search/spider.htmltext/..HideMe.mobi Browser Mozilla/5.0 (compatible; Baiduspider/2.0; url)
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexNewslinks; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Nano; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
 help.yahoo.comtext/..Mozilla/5.0 (YahooYSMcm/3.0.0; url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url)
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
 corp.naver.jp/text/..Mozilla/5.0 (compatible; NaverJapan/1.0; url)
msn
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url)
 search.msn.com/msnbot.htm-msnbot/0.01 (url)
 search.msn.com/msnbot.htm-msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url)
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.html-Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/-SeznamBot/3.0 (url)
discoveryengine
 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.html-Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.htmlimage/..Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.htmlNONE/wiktionary- Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.htmlapplication/oggMozilla/5.0 (compatible; discobot/2.0; url)
cibra
 cibra.de/text/..CiBra Data Collector (url)
soso
 help.soso.com/webspider.htmtext/..Sosospider(url)
 help.soso.com/webspider.htm-Sosospider(url)
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
 pear.php.net/text/..PEAR HTTP_Request class ( url )
 pear.php.net/image/..PEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8
 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.14
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url)
wordpress
 kterrl.wordpress.comtext/..WordPress/3.5-alpha; url
 klausgauger.wordpress.comtext/..WordPress/3.5-alpha; url
 dailyddose.wordpress.comtext/..WordPress/3.5-alpha; url
 thaiintelligentnews.wordpress.comtext/..WordPress/3.5-alpha; url
 josefboberg.wordpress.comtext/..WordPress/3.5-alpha; url
 theorbo1.wordpress.comtext/..WordPress/3.5-alpha; url
 02varvara.wordpress.comtext/..WordPress/3.5-alpha; url
 antinewworldorderparty.wordpress.comtext/..WordPress/3.5-alpha; url
 greatriversofhope.wordpress.comtext/..WordPress/3.5-alpha; url
 lobbyistsofficesofgrw.wordpress.comtext/..WordPress/3.5-alpha; url
 condette.wordpress.comtext/..WordPress/3.5-alpha; url
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url)
 en.wikipedia.org/wiki/User_talk:Blevintrontext/..BlevintronBot version 2012-05-19 contact url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
 en.wikipedia.orgtext/..url
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.18.0 url
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/image/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 toolbar.youdao.com/image/..Youdao Toolbar (url)
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ),gzip(gfe) (via translate.google.com)
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url)
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url)
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou inst spider/4.0(url)
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.38-15-generic; java 1.6.0_22; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_33; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-25-generic; java 1.6.0_24; Indian/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.12-gentoo; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0; Australia/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.32-vs2.3.2.3-dq67sw; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.3.4-2-ARCH; java 1.6.0_18; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
 yacy.net/bot.html-yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
 toolserver.org/~dispenser/text/..DispensersTools (url)
 toolserver.org/~dispenser/application/jsonDispensersTools (url)
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02
 toolserver.org/~dispenser/application/jsonPurge-script/1.0 (url)
yioop
 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url)
 www.yioop.com/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url)
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url)
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url)
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.htmlimage/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url)
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
 www.archive.orgimage/..Mozilla/5.0 (compatible; heritrix/3.1.0 url)
 www.archive.orgtext/..Mozilla/5.0 (compatible; heritrix/3.1.0 url)
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url)
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)
wikidict
 www.wikidict.detext/..url
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
 enwp.org/User:KingpinBottext/..KingpinBot (url)
 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url
sf
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
dataparksearch
 dataparksearch.org/bottext/..DataparkSearch/4.54-26052011 (url)
robdns
 robdns.comtext/..RobDNS Wiki rank indexer (url)
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url)
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url)
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url)
 www.gnip.com/-UnwindFetchor/1.0 (url)
 www.gnip.com/image/..UnwindFetchor/1.0 (url)
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
 github.com/NeilCrosby/wikislurpapplication/vnd.php.serializedWikiSlurp (url)
 github.com/edsu/wikitweetsapplication/jsonwikitweets <url
cognarius
 cognarius.comapplication/jsonAppsArlak/1.0 (url)
 cognarius.comtext/..AppsArlak/1.0 (url)
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
avantbrowser
 www.avantbrowser.comtext/..Advanced Browser (url)
 www.avantbrowser.comtext/..Avant Browser (url)
newsgator
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address )
whatrhymeswith
 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
feedshow
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
 www.feedshow.comtext/..FeedshowOnline (url)
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B206f); url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Mac OS X; 11.4.0 x86_64, Version 10.7.4 (Build 11E53); url)
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url)
bibalex
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url)
freebase
 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
wikiglass
 wikiglass.comtext/..url : mail address
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url)
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url)
proximic
 www.proximic.comtext/..Mozilla/5.0 (compatible; proximic; url)
commoncrawl
 www.commoncrawl.org/bot.htmltext/..CCBot/1.0 (url)
drupal
 drupal.org/text/..Drupal (url)
 drupal.org/text/..User-Agent: Drupal (url)
 drupal.org/image/..Drupal (url)
 drupal.org/application/xmlDrupal (url)
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
 SearchNearMe.com/contact.phptext/..SearchNearMe (url)
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.html-Mozilla/5.0 (compatible; bnf.fr_bot; url)
mysite
 www.mysite.comtext/..Mozilla/5.0 (compatible; myAbstractCrawler url)
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
graemef
 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
kula
 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
blogbridge
 www.blogbridge.com/text/..BlogBridge 2.13 (url)
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
orcabrowser
 www.orcabrowser.comtext/..Orca Browser (url)
nemui
 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
winpodder
 winpodder.comtext/..WinPodder (url)
ponderer
 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
zipcommander
 www.zipcommander.com/text/..1st ZipCommander (Net) - url
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
embed
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url)
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url)
seebot
 seebot.orgtext/..Lynx/2.8 (;url)
rssreader
 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
zootycoon
 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
snarfware
 www.snarfware.com/text/..Snarfer/0.x.x (url)
ac
 yazduni.ac.irtext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.clips.ua.ac.be/pages/patternapplication/jsonPattern/2.3 url
 nlp.ist.i.kyoto-u.ac.jp/text/..Kyoto-Crawler/2.0 (Mozilla-compatible; mail address (dot)kyoto-u(dot)ac(dot)jp; url)
it-influentials
 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
rssbandit
 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
feeds4all
 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
timewe
 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
plagger
 plagger.org/text/..Plagger/0.x.xx (url)
ranchero
 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
kr:6600
 www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsptext/..url
tumblr
 benderthewebrobot.tumblr.comtext/..Mozilla/5.0 (compatible; Bender; url)
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
neofonie
 spider.neofonie.detext/..MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to mail address ; url; mail address )
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/3.3 (Mac OS X; url; gzip-happy)
grid-son
 grid-son.comapplication/jsonurl
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
 spinn3r.com/robot-Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
weblio
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
superfeedr
 superfeedr.comapplication/xmlSuperfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
suggy
 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
mobileproxy
 mobileproxy.mobitext/..Mozilla/5.0 (compatible; MobileSurf; url)
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url)
 duckduckgo.com/duckduckpreview.htmltext/..DuckDuckPreview/1.0; (url)
 duckduckgo.com/duckduckpreview.html-DuckDuckPreview/1.0; (url)
whstour
 whstour.com/tokyotext/..WordPress/3.3.1; url
 whstour.com/osakatext/..WordPress/3.3.1; url
 whstour.com/nagoyatext/..WordPress/3.3.1; url
fotopedia
 www.fotopedia.comapplication/jsonPicor (url)
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
 www.sonyericsson.com/UAprof/R800xR301.xmltext/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address )
Anonymouse
 Anonymouse.org/image/..url (Unix)
 Anonymouse.org/text/..url (Unix)
netvibes
 www.netvibes.comtext/..Netvibes (url)
htw-berlin
 www.htw-berlin.detext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.htw-berlin.de-Mozilla/5.0 (compatible; heritrix/1.14.4 url)
acoon
 www.acoon.de/robot.asptext/..Mozilla/5.0 (compatible; AcoonBot/4.11.0; url)
my_website
 my_website.com/my_infopage.htmltext/..Mozilla/5.0 (compatible; heritrix/1.12.1 url)
 my_website.com/my_infopage.htmlimage/..Mozilla/5.0 (compatible; heritrix/1.12.1 url)
wocodi
 www.wocodi.com/crawlertext/..WocBot/Mozilla/5.0 (Wocodi Web Crawler 1.0; url; mail address )
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
ephorus
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.3.0; url)
pinterest
 pinterest.com/image/..Pinterest/0.1 url
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
setooz
 www.setooz.com/oozbot.htmltext/..Setooz/0.20 ( -- ; url ; mail address )
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot url)
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url)
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url)
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
thebestexercisesreview
 thebestexercisesreview.comtext/..WordPress/3.3.2; url
memidex
 www.memidex.com/_bottext/..Mozilla/5.0 (compatible; Memibot/1.0; url )
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
edu
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url)
froute
 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.4 (url)
 www.backgroundswitcher.com/text/..John's Background Switcher 4.4 (url)
ibis
 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
networkedblogs
 www.networkedblogs.comimage/..NetworkedBlogs (url;) AppEngine-Google; (http://code.google.com/appengine; appid: s~networkedblogshr)
108606.849999993total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0
 application/json
 application/xml
 text/..
 -
 application/x-www-form-urlencoded
 image/..
GoogleBot-Image/1.0
 image/..
 text/..
 -
MoovidaBot/0.1
 text/..
MediaWikiCrawler-Google/2.0 ( mail address )
 text/..
 -
php wikibot classes
 application/vnd.php.serialized
 text/..
 -
LinkParser/2.0
 text/..
GoogleBot-Image/1.0
 text/..
 image/..
 -
 application/json
Peachy MediaWiki Bot API Version 1.0
 application/vnd.php.serialized
wikiwix-bot-3.0
 text/..
 -
gsa-crawler (Enterprise; T2-DS3YYS6PYJWAS; mail address )
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
 text/..
 -
spider
 application/vnd.php.serialized
 text/..
 application/yaml
 image/..
 application/json
ClueBot/1.1
 application/vnd.php.serialized
 text/..
Answersbot
 text/..
 -
Pywikipediabot/2.0
 application/json
HTMLParser/1.6
 text/..
 image/..
Mozilla 5.0 (Apibot 0.32)
 application/vnd.php.serialized
Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
 text/..
 image/..
 application/vnd.php.serialized
ClueBot/2.0
 application/vnd.php.serialized
wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
 text/..
MediaWiki::Bot/1.00
 text/..
 application/json
 -
DigitalsmithsBot
 text/..
plantspedia data crawler
 text/..
mail address
 application/vnd.php.serialized
 application/json
 text/..
cleaner-wikipedia bot / self.maluke.com
 text/..
 application/json
GoogleBot 2.1
 text/..
Wikipath Bot (email: mail address )
 application/json
WikiBot/0.1
 text/..
 application/xml
 image/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
 application/json
MediaWiki::Bot/3.2.6
 application/json
 -
 text/..
DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 image/..
Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; mail address )
 text/..
 -
 image/..
FAST Search Web Crawler 14.0.0325.0000
 text/..
 -
 application/xml
 application/rsd+xml
python-wikitools/1.2 (User:BernsteinBot)
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
 image/..
 text/..
 application/json
 -
iXcrawler/0.1
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/x-www-form-urlencoded
 application/xml
 -
DotNetWikiBot/2.100 (Unix 2.6.32.38; )
 text/..
WikiCatResearchBot ( mail address )
 text/..
Tawbot (public svn release; plwiki)
 text/..
Web Crawler
 text/..
 -
Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
 text/..
AdMedia bot
 text/..
 image/..
 -
FAST Enterprise Crawler 6 used by Wipro Ltd ( mail address )
 text/..
 -
LinksCrawler 0.1beta
 text/..
 -
 image/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
SineBot/1.5.19(User:SineBot)
 application/vnd.php.serialized
 text/..
Webwiki Search Engine Bot - www.webwiki.de
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
 application/json
 application/x-www-form-urlencoded
Test Webbot
 text/..
Future GOCEBot - inquire of Allens for info
 application/json
 -
DotNetWikiBot/2.100 (Unix 5.10.0.0; )
 text/..
 application/xml
Amerla Search Bot V1.0 www.amerla.com
 text/..
 -
UCMore Crawler App
 text/..
 -
MLBot (www.metadatalabs.com/mlbot)
 text/..
 application/vnd.php.serialized
 image/..
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
 text/..
 -
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
 text/..
 -
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
 application/json
SU Nutch Spider/Nutch-1.4
 text/..
 -
 image/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/x-www-form-urlencoded
 application/xml
GermCrawler
 application/json
 text/..
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
 text/..
 -
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
 application/json
MediaWiki::Bot/3.005002
 application/json
 text/..
CorenSearchBot/1.5 en libwww-perl/6.02
 text/..
A .NET Web Crawler
 text/..
Future GOCEBot (v0.1) - inquire of Allens for info
 application/json
wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
 application/json
 text/..
SurakWare MediaWiki Bot/1.0
 text/..
SiocWikiBot/1.0
 application/vnd.php.serialized
 text/..
 -
Twitterbot/1.0
 text/..
 image/..
 -
Joycrawler Robot
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 -
 application/xml
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
 application/json
 text/..
MetallmanulBot for Wiktionary (run by Metallmanul)
 application/json
 text/..
wikiparser/1 CFNetwork/454.12.4 Darwin/10.8.0 (x86_64) (MacPro5,1)
 image/..
 text/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
TVersity Media Robot
 text/..
DNSTallyKwBot/0.2
 text/..
TrueKnowledgeBot bot mail address >
 application/vnd.php.serialized
 application/xml
 -
 text/..
XLinkBot/1.00
 text/..
COIBot/1.00
 text/..
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
 text/..
AniBot/0.9 php/curl
 application/vnd.php.serialized
UniversalFeedParser/5.1.1 https://code.google.com/p/feedparser/
 text/..
 application/xml
 -
 application/x-external-editor
HosiryuhosiBot IRC-RecentChanges Util
 -
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
OrlodrimBot/1.0
 text/..
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
 text/..
 image/..
infraEnterprise v8 Web Crawler
 -
MyCuteBot/0.1
 text/..
 application/json
 application/vnd.php.serialized
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
 image/..
 text/..
HRoestBot, de-wikipedia using pywikipedia framework
 text/..
 application/json
bitlybot
 text/..
 image/..
 -
 application/pdf
HTMLParser/2.0
 text/..
 image/..
 -
LauschenBot/1.0 ( mail address )
 text/..
COIBot/2.0
 text/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.0/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 368 subscribers; feed-id=3852576738117026533)
 application/xml
 -
LonnyLotBot/HumansDidWhatv1
 text/..
 -
 application/ogg
Vineet's BOT
 text/..
DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
YBot/0.1
 application/vnd.php.serialized
SchoolReviewNetworkWikiBot
 application/json
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
 application/json
 text/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.1.0/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
 application/xml
 text/..
Empedia Bot
 text/..
 -
DotNetWikiBot/2.100 (Unix 3.0.0.12; )
 text/..
 application/xml
FAST Enterprise Crawler/5.3.4 ( mail address )
 text/..
 -
DotNetWikiBot/2.96 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
 application/xml
My Bot
 text/..
 image/..
Xaldon WebSpider 2.7.b8
 text/..
OrangeCrawler/Nutch-1.0 ( mail address )
 text/..
TestCrawler
 text/..
wordcrawler/1.0
 text/..
TwynCatBot/0.1 (Contact: www.twyn.com)
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
 application/json
super cool bot
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (compatible; UnisterBot; mail address )
 text/..
My Nutch Spider/Nutch-1.4
 text/..
CaBot Script (running on nightshade.toolserver.org)
 application/vnd.php.serialized
HBC Archive Indexerbot 0.9a
 text/..
SINA_ROBOT; Mozilla/5.0 (Windows; Windows NT 5.1; MSIE8.0; zh-CN; rv:1.9.1.8) Gecko/20100202 Firef8
 text/..
 image/..
Mozilla/5.0 (iPhone; CPU iPhone OS 4_0_1 like Mac OS X; fr-fr) OrangeBot-Mobile AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8A306 Safari/( mail address )
 image/..
 text/..
 -
python-wikitools/1.2 (User:LaraBot)
 application/json
Mozilla/5.0 crawler/suggest.io
 text/..
Spinuf Spider
 text/..
SxBot/V-1.0
 text/..
 -
AsgardBot - DotNetWikiBot/2.100 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
Goalkeeperbot(User:Beetstra)/1.0
 text/..
Phantom.js bot
 image/..
 text/..
FAST Enterprise Crawler/6.7.8 ( mail address )
 text/..
 -
Mozilla 5.0 (Apibot 0.30b5)
 application/vnd.php.serialized
GoogleBot
 text/..
 image/..
 -
Mozilla/5.0 (Bgbot 0.5)
 text/..
iNextWeb Crawler v.1.1- email: mail address
 text/..
WorldOfMusicBetaBot/1.0
 text/..
wAPI/1.1 (Bot: NoomBot Operator: Noommos Contact: mail address )
 application/vnd.php.serialized
Geni ircpybot 1.0
 text/..
 application/json
DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
MediaWiki::Bot/3.1.6 (User:SporkBot)
 application/json
Zing-BottaBot/2.0
 text/..
 -
MediaWiki::Bot 3.1.5
 application/json
php WalkingSoulBot
 application/vnd.php.serialized
 application/json
RelationsBot/1.0
 text/..
 application/json
wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0
 image/..
 application/json
 text/..
wikbot/1.60 CFNetwork/485.12.7 Darwin/10.4.0
 image/..
 text/..
 application/json
JavaCrawler/1.1
 text/..
20726.62total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, Aug 10, 2012 12:12
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers