Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Apr 2012 - 30 Apr 2012 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 68,092,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 460,381,230 external requests, which is 14.8%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~senchaiosrc)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cloudcrawling)
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat)
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
 code.google.com/appengine-AppEngine-Google; (url; appid: s~senchaiosrc)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
 code.google.com/appenginetext/..Wiki.java 0.25 AppEngine-Google; (url; appid: wikipediatools)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: myproxywx)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 code.google.com/appengineapplication/jsonMozilla 4.0 AppEngine-Google; (url; appid: prfleme)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..crawlr AppEngine-Google; (url; appid: s~google.com:crawlr-staging)
 code.google.com/appengine-AppEngine-Google; (url; appid: s~rain-soul)
 code.google.com/p/crawler4j/text/..crawler4j (url)
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy)
 code.google.com/p/rondaapplication/jsonRonda - url
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~deutiki)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp6)
 code.google.com/appenginetext/..Mozilla/5.0 AppEngine-Google; (url; appid: s~app3123)
 docs.google.com-Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: prfleme)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wagagate)
 code.google.com/appengine-AppEngine-Google; (url; appid: s~rain-soul2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy2031)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: misc-tools)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openeyeproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100thpriest)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp8)
 code.google.com/appengineimage/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cmd-proxy)
 code.google.com/p/crawler4j/-crawler4j (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bie99miracle)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web4proxy)
 code.google.com/appengine-AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nashimlive-nashimnx)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jptaravellahighschool)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kyaysarlay)
 docs.google.com-Mozilla/5.0 (compatible; GoogleDocs; documents; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nation4india)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kaveriselvaraj)
 code.google.com/appengine-AppEngine-Google; (url; appid: s~rain-soul4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pox)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivankrisproxyserver)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxynaungnaung)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn03)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 114proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools6)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: prexytwo)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: argim-free)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy)
 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: vipoembed)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tinkernutsearch)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; legacyeditor; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: prexyproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cdeskinsp)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 6.1; zh-CN; rv:1.9.2.2) AppEngine-Google; (url; appid: fwall-w15)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: betafxserver)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cachehew)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps005)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxies-list2)
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 developers.facebook.comimage/..facebookplatform/1.0 (url)
 developers.facebook.com-facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url)
 developers.facebook.comtext/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.phpapplication/vnd.php.serializedfacebookexternalhit/1.1 (url)
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmapplication/vnd.php.serializedMozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b4
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html))
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1 url)
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html-'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
 www.baidu.com/search/spider.htm-Baiduspider-image(url)
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: misc-tools)
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexNewslinks; url)
 yandex.com/botsapplication/vnd.php.serializedMozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url)
msn
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
 search.msn.com/msnbot.htm-msnbot/2.0b (url)
 search.msn.com/msnbot.htm-msnbot/2.0b (url)._
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
 wwwgogetpapers.com/text/..User-Agent: GoGetPapersBot (url)
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.html-Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/-SeznamBot/3.0 (url)
cibra
 cibra.de/text/..CiBra Data Collector (url)
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
 pear.php.net/text/..PEAR HTTP_Request class ( url )
 pear.php.net/image/..PEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.14
 pear.php.net/-PEAR HTTP_Request class ( url )
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/image/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 toolbar.youdao.com/image/..Youdao Toolbar (url)
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 www.youdao.com/help/webmaster/spider/application/vnd.php.serializedMozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
dataparksearch
 dataparksearch.org/bottext/..DataparkSearch/4.54-26052011 (url)
wordpress
 driwancybermuseum.wordpress.comtext/..WordPress/3.4-alpha-20343; url
 omadeon.wordpress.comtext/..WordPress/3.4-beta1; url
 klima47.wordpress.comtext/..WordPress/3.4-beta1; url
 driwancybermuseum.wordpress.comtext/..WordPress/3.4-beta2-20460; url
 klausgauger.wordpress.comtext/..WordPress/3.4-beta1; url
 klima47.wordpress.comtext/..WordPress/3.4-beta3-20603; url
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07application/vnd.php.serializedSogou web spider/4.0(url)
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.9; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_30; Asia/ja) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.42.12-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/ro) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.42.9-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
 yacy.net/bot.html-yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.9; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-16-server; java 1.6.0_23; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/fi) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-5-486; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.0.0-19-generic-pae; java 1.7.0_147-icedtea; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo; java 1.6.0_31; Europe/en) url
soso
 help.soso.com/webspider.htmtext/..Sosospider(url)
 help.soso.com/webspider.htm-Sosospider(url)
 help.soso.com/webspider.htmapplication/xmlSosospider(url)
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
 en.wikipedia.org/text/..CSResearch/Nutch-1.2 (Research - Natural Language Engineering and Web Applications; url; mail address )
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.18.0 url
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19 url
 en.wikipedia.orgtext/..url
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
entireweb
 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
discoveryengine
 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.htmlimage/..Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.html-Mozilla/5.0 (compatible; discobot/2.0; url)
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url)
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
 www.archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; archive.org_bot url)
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url)
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
 toolserver.org/~dispenser/text/..DispensersTools (url)
 toolserver.org/~dispenser/application/jsonDispensersTools (url)
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url)
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url)
 www.gnip.com/-UnwindFetchor/1.0 (url)
 www.gnip.com/image/..UnwindFetchor/1.0 (url)
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
enotes
 www.enotes.comtext/..eNotesBot 2.0 (url)
 www.enotes.comimage/..eNotesBot 2.0 (url)
 www.enotes.com-eNotesBot 2.0 (url)
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url)
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url)
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
 github.com/NeilCrosby/wikislurpapplication/vnd.php.serializedWikiSlurp (url)
 github.com/edsu/wikitweetsapplication/jsonwikitweets <url
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
wikidict
 www.wikidict.detext/..url
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
 enwp.org/User:KingpinBottext/..KingpinBot (url)
 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
patriotradionetwork
 www.patriotradionetwork.comtext/..WordPress/3.3.1; url
 www.patriotradionetwork.comtext/..WordPress/3.3.2; url
sf
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
ephorus
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.2.2; url)
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url)
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (iOS; 11.0.0 AppleTV2,1, Version 5.0.1 (Build 9A406a); url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1 (Build 9B179b); url)
 www.xbmc.org-XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
 www.xbmc.orgimage/..XBMC/11.0-RC2 Git:20120229-f38655f (iOS; 11.0.0 AppleTV2,1, Version 5.0.1 (Build 9A406a); url)
zeebox
 www.zeebox.comapplication/jsonZeebox (url)
avantbrowser
 www.avantbrowser.comtext/..Advanced Browser (url)
 www.avantbrowser.comtext/..Avant Browser (url)
feedshow
 www.feedshow.comtext/..FeedshowOnline (url)
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
newsgator
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
wikiglass
 wikiglass.comtext/..url : mail address
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address )
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
 SearchNearMe.com/contact.phptext/..SearchNearMe (url)
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
freebase
 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address )
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url)
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.html-Mozilla/5.0 (compatible; bnf.fr_bot; url)
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
whatrhymeswith
 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
kr:6600
 www.checkprivacy.or.kr:6600/RS/PRIVACY_FAQ.jsptext/..url
 www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsptext/..url
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url)
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
kalooga
 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url)
 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url)
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url)
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url)
whstour
 whstour.com/tokyotext/..WordPress/3.3.1; url
 whstour.com/osakatext/..WordPress/3.3.1; url
 whstour.com/nagoyatext/..WordPress/3.3.1; url
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
zootycoon
 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
graemef
 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
timewe
 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
zipcommander
 www.zipcommander.com/text/..1st ZipCommander (Net) - url
feeds4all
 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
snarfware
 www.snarfware.com/text/..Snarfer/0.x.x (url)
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
seebot
 seebot.orgtext/..Lynx/2.8 (;url)
blogbridge
 www.blogbridge.com/text/..BlogBridge 2.13 (url)
it-influentials
 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
orcabrowser
 www.orcabrowser.comtext/..Orca Browser (url)
kula
 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
rssbandit
 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
ranchero
 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
rssreader
 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
nemui
 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
metamoji
 www.metamoji.com/jp/crawler.htmltext/..Mozilla/5.0 (compatible; MetamojiCrawler/1.0; url
winpodder
 winpodder.comtext/..WinPodder (url)
plagger
 plagger.org/text/..Plagger/0.x.xx (url)
ponderer
 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address )
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
superfeedr
 superfeedr.comapplication/xmlSuperfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
tiscali
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.01.18 url)
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
drupal
 drupal.org/text/..User-Agent: Drupal (url)
 drupal.org/text/..Drupal (url)
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/3.3 (Mac OS X; url; gzip-happy)
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url)
 duckduckgo.com/duckduckpreview.html-DuckDuckPreview/1.0; (url)
 duckduckgo.com/duckduckpreview.htmltext/..DuckDuckPreview/1.0; (url)
weblio
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
embed
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url)
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; url)
govid
 govid.mobi/bot.phptext/..Mozilla/5.0 (compatible; gofind; url)
scoutjet
 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
netvibes
 www.netvibes.comtext/..Netvibes (url)
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
Anonymouse
 Anonymouse.org/image/..url (Unix)
 Anonymouse.org/text/..url (Unix)
zapbot
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url)
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url)
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url)
tumblr
 benderthewebrobot.tumblr.comtext/..Mozilla/5.0 (compatible; Bender; url)
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
pinterest
 pinterest.com/image/..Pinterest/0.1 url
 pinterest.com/-Pinterest/0.1 url
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
grid-son
 grid-son.comapplication/jsonurl
froute
 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
proximic
 www.proximic.comtext/..Mozilla/5.0 (compatible; proximic; url)
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
 www.sonyericsson.com/UAprof/R800xR301.xml-Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
netarkivet
 netarkivet.dk/website/info.htmlimage/..Mozilla/5.0 (compatible; heritrix/1.12.1b url)
 netarkivet.dk/website/info.htmltext/..Mozilla/5.0 (compatible; heritrix/1.12.1b url)
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
sixxs
mondowindow
 www.mondowindow.comtext/..MondoWindow (url)
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url)
edu
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url)
newpureglobe
 www.sports.newpureglobe.comtext/..WordPress/3.3.1; url
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
104713.469999993total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0
 application/json
 application/xml
 text/..
 -
 image/..
 application/ogg
GoogleBot-Image/1.0
 text/..
 image/..
 -
MediaWikiCrawler-Google/2.0 ( mail address )
 text/..
 -
php wikibot classes
 application/vnd.php.serialized
 text/..
MoovidaBot/0.1
 text/..
LinkParser/2.0
 text/..
 -
GoogleBot-Image/1.0
 text/..
 image/..
 -
 application/vnd.php.serialized
 application/json
Peachy MediaWiki Bot API Version 1.0
 application/vnd.php.serialized
 -
 text/..
wikiwix-bot-3.0
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
 text/..
 -
 video/ogg
SemrushBot/0.92
 text/..
 application/ogg
 image/..
 -
 audio/midi
ClueBot/2.0
 application/vnd.php.serialized
Answersbot
 text/..
ClueBot/1.1
 application/vnd.php.serialized
mail address
 application/vnd.php.serialized
 text/..
 application/json
Pywikipediabot/2.0
 application/json
spider
 text/..
 application/vnd.php.serialized
 application/json
 image/..
 -
Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
 text/..
 application/vnd.php.serialized
 -
 image/..
 application/xml
gsa-crawler (Enterprise; T2-DS3YYS6PYJWAS; mail address )
 text/..
 -
 image/..
Mozilla 5.0 (Apibot 0.32)
 application/vnd.php.serialized
K-Crawler
 text/..
 application/opensearchdescription+xml
 application/rsd+xml
 -
 image/..
 application/ogg
wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
GSLFbot
 text/..
 image/..
 application/xml
 application/vnd.php.serialized
 -
AnomieBOT 1.0 (TagDater)
 application/json
 application/x-www-form-urlencoded
DigitalsmithsBot
 text/..
BritannicaProjBot mail address
 text/..
DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 image/..
 application/ogg
DotNetWikiBot/2.97 (Unix 2.6.32.38; )
 text/..
 application/xml
MediaWiki::Bot/3.2.6
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
 text/..
bob's crappy crawler; contact: mail address
 text/..
plantspedia data crawler
 text/..
python-wikitools/1.2 (User:BernsteinBot)
 application/json
 application/x-www-form-urlencoded
 text/..
TrueKnowledgeBot bot mail address >
 application/xml
 application/vnd.php.serialized
GoogleBot 2.1
 text/..
 image/..
Tawbot (public svn release; plwiki)
 text/..
Test Webbot
 text/..
MediaWiki::Bot/1.00
 text/..
 application/json
DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
MLBot (www.metadatalabs.com/mlbot)
 text/..
 application/vnd.php.serialized
 -
 image/..
SineBot/1.5.18(User:SineBot)
 application/vnd.php.serialized
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer)
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
 image/..
 text/..
 -
SchoolReviewNetworkWikiBot
 application/json
FAST Enterprise Crawler 6 used by ESP ( mail address )
 text/..
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
 text/..
 -
Shad robot/1.0
 text/..
 -
AniBot/0.9 php/curl
 application/vnd.php.serialized
 -
UCMore Crawler App
 text/..
 -
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
 text/..
 -
JavaCrawler/1.1
 text/..
 -
 image/..
DotNetWikiBot/2.97 (Unix 5.10.0.0; )
 application/xml
 text/..
AnomieBOT 1.0 (FlagIconRemover)
 application/json
COIBot/1.00
 text/..
CorenSearchBot/1.5 en libwww-perl/6.02
 text/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.4.121) Opera Mini/3.1
 image/..
 -
 text/..
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
 text/..
 -
MireoBot
 text/..
 -
cis455crawler
 text/..
 -
 image/..
DotNetWikiBot/2.100 (Unix 2.6.32.38; )
 text/..
DNSTallyKwBot/0.2
 text/..
Webwiki Search Engine Bot - www.webwiki.de
 text/..
TheKeens bot
 text/..
GermCrawler
 application/json
 text/..
SiocWikiBot/1.0
 application/vnd.php.serialized
 text/..
DotNetWikiBot/2.96 (Unix 5.10.0.0; )
 text/..
 application/xml
AnomieBOT 1.0 (TemplateSubster)
 application/json
DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
AnomieBOT 1.0 (PERTableUpdater)
 application/json
 text/..
OpenSearchServer_Bot
 text/..
CyberfoxBot
 text/..
 -
 image/..
Mozilla/5.0 (compatible; BeetleBot; )
 text/..
 image/..
 -
wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
Twitterbot/1.0
 text/..
 image/..
 -
XLinkBot/1.00
 text/..
SurakWare MediaWiki Bot/1.0
 text/..
 application/xml
YBot/0.1
 application/vnd.php.serialized
TigrinyaCrawler( mail address )
 text/..
 -
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
 text/..
HosiryuhosiBot IRC-RecentChanges Util
 text/..
 -
FAST Enterprise Crawler 6 used by LexisNexis ( mail address )
 text/..
 -
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 application/x-www-form-urlencoded
HTMLParser/1.6
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
OrlodrimBot/1.0
 text/..
SearchBot
 text/..
RobBot ( mail address )
 application/vnd.php.serialized
HTMLParser/2.0
 text/..
AnomieBOT 1.0 (BAGBot)
 application/json
 text/..
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
 image/..
 -
 text/..
TVersity Media Robot
 text/..
super cool bot
 application/vnd.php.serialized
HRoestBot, de-wikipedia using pywikipedia framework
 text/..
 application/json
GNAA-bot
 text/..
COIBot/2.0
 text/..
Bot work. [[no:User:PladaskBot]].
 application/vnd.php.serialized
wikbot/1.50 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 -
 text/..
wikbotlite/1.50 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
Freebase Deathbot
 text/..
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 369 subscribers; feed-id=3852576738117026533)
 application/xml
 -
DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
bitlybot
 text/..
 -
 image/..
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
 application/xml
 text/..
FAST Enterprise Crawler/5.3.4 ( mail address )
 text/..
 -
AnomieBOT 1.0 (ReplaceExternalLinks2)
 application/json
wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0
 image/..
 application/json
 -
 text/..
Empedia Bot
 text/..
OrangeCrawler/Nutch-1.0 ( mail address )
 text/..
Goalkeeperbot(User:Beetstra)/1.0
 text/..
DotNetWikiBot/2.98 (Unix 3.0.0.12; )
 text/..
 application/xml
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
MSRBOT
 text/..
DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; mail address )
 text/..
 application/pdf
HAZY.SPIDER/Nutch-1.4
 text/..
 -
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
 application/xml
Wikibot/1.56 CFNetwork/520.3.2 Darwin/11.3.0 (x86_64) (MacBookPro8,1)
 -
 image/..
 application/json
 text/..
WikiCatResearchBot ( mail address )
 text/..
AnomieBOT 1.0 (RandomPagePicker)
 application/json
MyCuteBot/0.1
 text/..
 application/json
 application/vnd.php.serialized
wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
Applied-Technologies-Inc-Spider/Nutch-1.4
 text/..
Local Site Parser 1.0
 text/..
My Nutch Spider/Nutch-1.4
 text/..
Markus Möller's super cool bot
 text/..
CheMoBot/1.00
 text/..
AnomieBOT 1.0 (AFDMergeFromCleaner)
 application/json
AnomieBOT 1.0 (DeletionSortingCleaner)
 application/json
python-wikitools/1.2 (User:LaraBot)
 application/json
WikiBot/0.1
 text/..
Geni ircpybot 1.0
 text/..
 application/json
 application/xml
DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
gsa-crawler (Enterprise; M2-PHC4GAAC6LSJA; mail address )
 text/..
Mozilla/5.0 (Bgbot 0.5)
 text/..
Zing-BottaBot/1.0
 text/..
AnomieBOT 1.0 (NewArticleAFDTagger)
 application/json
 application/x-www-form-urlencoded
Nutch-Spider/Nutch-1.4
 text/..
FUB-RSS-standalone-crawler/Nutch-1.3
 text/..
Baiduspider
 text/..
ClueBot/2.0 (ClueBot NG Report Interface)
 text/..
IssueCrawler
 text/..
Mozilla 5.0 (Apibot 0.30b5)
 application/vnd.php.serialized
HBC Archive Indexerbot 0.9a
 text/..
TestCrawler
 text/..
DotNetWikiBot/2.9 (Unix 5.10.0.0; )
 text/..
gsa-crawler (Enterprise; M2-K3NP42JC6JWAS; mail address )
 text/..
 application/pdf
FAST Enterprise Crawler/6.7.8 ( mail address )
 text/..
 -
cleaner-wikipedia bot / self.maluke.com
 text/..
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
KD Bot
 text/..
 -
 image/..
merocrawl/Nutch-1.4 (merobase crawler; www.merobase.com; mail address )
 text/..
 application/pdf
 -
 image/..
GoogleBot
 text/..
 image/..
DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
 text/..
18421.78total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, Aug 10, 2012 12:09
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers