Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Apr 2013 - 3 Apr 2013 (last 12 months)
000 ⇒ k
 

 This analysis is based on a 1:1000 sampled server log (squids)

Warning: all recent Wikimedia traffic analysis reports have been generated from old scripts.

The scripts are orphaned, and have not been maintained for at least 6 months. Many bugs are considerably older.
Known Bugzilla bugs: 46190, 46191, 46195, 46201, 46205, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 46289

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 97,238,670 page requests (mime type text/html only!) per day are considered crawler requests, out of 631,451,330 external requests, which is 15.4%

Page requests for crawlers that specify a url in the agent string
Count
x 1000
Secondary domain
(~site) name
URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kyouen3) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 code.google.com/appenginetext/..Acre/dev/53:893 staging.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sancampoen) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~batshiitinsane) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..Wiki.java 0.27 AppEngine-Google; (url; appid: wikipediatools) - -
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~italiatiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-api) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: surf6003) - -
 code.google.com/appenginetext/..Acre/dev/51:889 ubiquity.freebaseapps.com AppEngine-Google; (url; appid: s~freebaseapps) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ninjallamastuff) - -
 code.google.com/appengineapplication/jsonMozilla/5.0 AppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appengineapplication/jsonUser-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.56 Safari/536.5 AppEngine-Google; (url; appid: s~in-onda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khrixy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: vipoembed) - -
 www.google.com/coop/cse/creftext/..PageFetcher-Google-CoOp;((url) - -
 code.google.com/appenginetext/..Mozilla/5.0 AppEngine-Google; (url; appid: s~vodio-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~win8wale) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~theunblock) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..Python-urllib/2.7 AppEngine-Google; (url; appid: s~hr-pulsesubscriber) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beansacks) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: s~ninjallamastuff) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~batshiitinsane) - -
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) SitemapProbe - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxypython7) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)1 readers - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url) ja, *;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url) - -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..Y!J-BRU/VSIDX dead link checker (url) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; ) PHP/5.4 - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
soso
 help.soso.com/webspider.htmtext/..Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmapplication/xmlSosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
wordpress
 wildanrenaldi.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 williamstickevers.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 josefboberg.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 williamstickevers.wordpress.comimage/..WordPress/3.6-alpha-23334; url - -
 ledinobleu.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 bestofactus.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 ainhoaaristizabal.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 urielarte.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 cheltjules.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 klausgauger.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 escogitur.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 cedricgagneux.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 kalafudra.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 ramahanumanraksha.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 imcradiodotnet.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 armandecastro.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 foxhugh.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 euzicasa.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 whatownsme.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 kikestark.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 beyondthelies.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 02varvara.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 melusinefee.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 bharatabharati.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 gunnyg.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 energiaslibres.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 wildanrenaldi.wordpress.comimage/..WordPress/3.6-alpha-23334; url - -
 onceuponavoice.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 mcdens13.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
ac
 www.clips.ua.ac.be/pages/patterntext/..Pattern/2.3 url - -
 www.celese.sci.waseda.ac.jptext/..2QC (url; mail address ) - -
 www.clips.ua.ac.be/pages/patternapplication/jsonPattern/2.5 url - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/application/xmlAhrefsBot.Feeds v0.1; url - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
 crawler.sistrix.net/image/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17 - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) - -
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmlapplication/jsonMozilla/5.0 (compatible; JikeSpider; url) - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.20.0 url - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url - -
 en.wikipedia.org/wiki/Web_crawlertext/..Robobot/1.0 (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.20 url - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~dispenser/application/jsonDispensersTools (url) - -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug) - -
 toolserver.org/~platonides/catdown/image/..catdown High-resolution_TIFF_images_from_the_National_Archives_and_Records_Administration (url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
 www.gnip.com/image/..UnwindFetchor/1.0 (url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
yacy
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 2.6.23.17-dbserv; java 1.6.0_04; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.6.0_41; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-36-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; i386 Linux 3.8.0-8-generic; java 1.7.0_17; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Australia/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-39-generic; java 1.6.0_27; Europe/lv) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-46-server; java 1.6.0_26; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-39-generic; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-26-generic; java 1.6.0_27; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-19-pve; java 1.6.0_18; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-1.16-default; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.1; java 1.6.0_43; Asia/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-40-generic; java 1.7.0_15; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-40-generic; java 1.7.0_15; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-39-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
coccoc
 help.coccoc.com/text/..coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..coccoc/1.0 (url) - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
localhost
 localhost/wordpresstext/..WordPress/3.5.1; url - -
 localhost/yioop/bot.phptext/..Mozilla/5.0 (compatible; TESTROBOT; url) - -
 localhost/2text/..WordPress/3.5.1; url - -
 localhost/1text/..WordPress/3.5.1; url - -
 localhost/dizzyworldtext/..WordPress/3.5.1; url - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
wikidict
 www.wikidict.detext/..url - -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 goo.gl/7y4SXimage/..GoogleProducer; (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
frontle
 www.frontle.com/application/vnd.php.serializedFrontleBot/1.0 (url) - -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
plos
 alm.plos.orgapplication/jsonPLoS Article Level Metrics - url - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) af -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
elcidharth
 elcidharth.comtext/..WordPress/3.6-alpha-23334; url - -
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url) - -
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
 wikimpress.org/-Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) ja -
wikimedia
 commons.wikimedia.org/wiki/User:Thumbnails_Check_Botimage/..Thumbnails_Check_Bot/0.1 (url; beta) - -
 commons.wikimedia.org/wiki/User:Thumbnails_Check_Bottext/..Thumbnails_Check_Bot/0.1 (url; beta) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
bibalex
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url) en-us,en;q=0.5 -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
thearchangelmichael
 archives.thearchangelmichael.nettext/..WordPress/3.5.1; url - -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
cognarius
 cognarius.comapplication/jsonAppsArlak/1.0 (url) - -
 cognarius.comtext/..AppsArlak/1.0 (url) - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
wandex
 wandex.nettext/..Mozilla/5.0 (compatible; World Wide Web Wanderer (Wandex Bot)/1.4; url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
chickyrun
 chickyrun.tk/text/..ChickyBot/1.1 (url; mail address ) - -
toshiba
 www.toshiba.co.jp/rdc/about/crawl_info_en.htmtext/..TosCrawler/Nutch-1.6 (url; ' mail address dot co dot jp') en-us,en-gb,en;q=0.99,*;q=0.01 -
rebelmouse
 rebelmouse.comimage/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.comtext/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
sunaga-lab
 www.sunaga-lab.com/graham-bottext/..Grahambot/0.1 (url) - -
zibalinks
 www.zibalinks.comtext/..Ziba/Nutch-1.4 (Ziba Links Web Spider; url) en-us,en-gb,en;q=0.7,*;q=0.3 -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
xbmc
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
vermagerd
 www.vermagerd.be/wptext/..WordPress/3.4.2; url - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 wummel.github.com/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.4; url) - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 wiki.github.com/bixo/bixo/bixocrawlertext/..Mozilla/5.0 (compatible; pub-crawler; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
potaru
 potaru.com/robo.htmltext/..Mozilla/5.0 (compatible; Robo/1.0b; url)/Nutch-1.2 en-us,en-gb,en;q=0.7,*;q=0.3 -
sf
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url) en-us,en;q=0.5 -
 magpierss.sf.nettext/..MagpieRSS/0.7x (url) en-us,en;q=0.5 -
treycopplandmusic
 beatsblog.treycopplandmusic.comtext/..WordPress/3.5.1; url - -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
rockmelt
 rockmelt.comtext/..RockmeltEmbedder (url; mail address ) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X101.xmlimage/..url en,hi -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
queryseeker
 queryseeker.com/bot.htmltext/..QuerySeekerSpider ( url ) - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl-ja.htmltext/..ICC-Crawler (Mozilla-compatible; mail address ; url) ja -
mysite
 www.mysite.com/text/..MyBot/1.0 (url) - -
 www.mysite.com/application/jsonMyBot/1.0 (url) - -
mindano
 www.mindano.comimage/..Mindano (url) - -
n-grams
 www.n-grams.org/icorpusbot.htmltext/..iCorpusBot (url) es-es,en-us;q=0.7,en;q=0.3 -
simplepie
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
newsgator
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers) en-us,en;q=0.5 -
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP) en-us,en;q=0.5 -
zapbot
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url) - -
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url) - -
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
site-shot
 www.site-shot.com/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -
reget
 www.reget.comtext/..Mozilla/4.0 (compatible; url>ReGet Deluxe 5.1; Windows NT 6.0) - -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.1 (url) - -
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
backgroundswitcher
 www.backgroundswitcher.com/text/..John's Background Switcher 4.4 (url) - -
 www.backgroundswitcher.com/image/..John's Background Switcher 4.6 (url) - -
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url) en-us,en;q=0.5 -
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url) en-us,en;q=0.5 -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - -
w3
 validator.w3.org/servicestext/..W3C_Validator/1.3 url - -
nvdev
 qa.nvdev.comtext/..Netvibes (url) - -
 trunk.nvdev.comtext/..Netvibes (url) - -
website-datenbank
 www.website-datenbank.de/text/..netEstate NE Crawler (url) - -
weborking
 weborking.comtext/..Weborking(url) - -
 weborking.comapplication/jsonWeborking(url) - -
orcabrowser
 www.orcabrowser.comtext/..Orca Browser (url) en -
 www.orcabrowser.comtext/..Orca Browser (url) en-us,en;q=0.5 -
studiofaca
 www.studiofaca.com/text/..Mozila/5.0 (compatible; StudioFACA Search; url) - -
onemusicapi
 www.onemusicapi.comtext/..onemusicapi.com/beta url - -
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -
QtWeb
archive-org
 archive-org.com/botimage/..Mozilla/5.0 (compatible; archive-org.com/1.1; url) - -
 archive-org.com/bottext/..Mozilla/5.0 (compatible; archive-org.com/1.1; url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url) en-us -
feedshow
 www.feedshow.comtext/..FeedshowOnline (url) en-us,en;q=0.5 -
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber) en-us,en;q=0.5 -
scrapy
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
mobileproxy
 mobileproxy.mobitext/..Mozilla/5.0 (compatible; MobileSurf; url) - -
115354.920000002total

Page requests for probable crawlers, recognized by keyword
Count
x 1000
Agent string
  Mime type (count ≥ 3)
php wikibot classes - -
 application/vnd.php.serialized
 text/..
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
Peachy MediaWiki Bot API Version 2.0 (beta) - -
 application/vnd.php.serialized
 text/..
 image/..
enwiki_removal/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
LinkParser/2.0 - -
 text/..
spider - -
 text/..
 application/vnd.php.serialized
 application/json
pywikipedia-radeh.py/r11231 Pywikipediabot/1.0 - -
 application/json
 application/xml
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
PythonWikipediaBot/1.0 - -
 application/json
 application/xml
 text/..
 -
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
 application/xml
 application/opensearchdescription+xml
RobBot ( mail address ) - -
 application/vnd.php.serialized
Mozilla/5.0 (compatible; SearchBot) - -
 text/..
wikidata_create/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-wikidata_wd.py/r11181 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-wikidata9de.py/r11311 Pywikipediabot/1.0 - -
 application/json
 application/xml
DotNetWikiBot/2.101 (Unix 3.0.0.32; ) - -
 text/..
 application/xml
pywikipedia-hamsang.py/r11308 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
tigerbot - -
 application/json
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 image/..
pywikipedia-wikidata9de.py/r11228 Pywikipediabot/1.0 - -
 application/json
 application/xml
ClueBot/1.1 - -
 application/vnd.php.serialized
wikiwix-bot-3.0 - -
 text/..
DotNetWikiBot/2.101 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
pywikipedia-hamsang.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata9de.py/r11181 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_war.py/r11181 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
 application/xml
ClueBot/2.0 - -
 application/vnd.php.serialized
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
pywikipedia-wikidata_cleanlabel.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
TrueKnowledgeBot bot mail address > - -
 application/xml
 application/vnd.php.serialized
 image/..
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 image/..
pywikipedia-wikidata_wd.py/r11311 Pywikipediabot/1.0 - -
 application/json
 application/xml
process_baseball/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DigitalsmithsBot - -
 text/..
pywikipedia-wikidata_disamb.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11096 Pywikipediabot/1.0 - -
 application/xml
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
acGen-acGenerator_trunk.py/r11321 Pywikipediabot/1.0 - -
 application/json
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_wceb.py/r11181 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
 application/xml
trunk-featured.py/r11322 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla 5.0 (Apibot 0.32) - -
 application/vnd.php.serialized
maj_articles_recents/r11235 Pywikipediabot/2.0 - -
 application/json
MorbZ-Bot - -
 application/json
pywikipedia-wikidata9de.py/r11304 Pywikipediabot/1.0 - -
 application/json
UAA-UAA.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11308 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11301 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Bot/3.2.6 - -
 application/json
pywikipedia-commonscat_wikidata.py/r11104 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11322 Pywikipediabot/1.0 - -
 application/json
 application/xml
trunk-checkimages.py/r11300 Pywikipediabot/1.0 - -
 application/json
Semantix Bot 0.1 - -
 text/..
Wikipediabot-mywikidatabot2.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
pywikipedia-wikidata9.py/r11311 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-wdph25.py/r11275 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-iw2make.py/r11317 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-wdph22.py/r11275 Pywikipediabot/1.0 - -
 application/json
 application/xml
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 application/xml
Tawbot (public svn release; plwiki) - -
 text/..
Wikibot/2.0.2 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
Python27-pythonw.exe/r11204 Pywikipediabot/1.0 - -
 application/json
pywikipedia-newiw2.py/r11321 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-wdph23.py/r11275 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pywikipedia-featured.py/r11307 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
mrajedrez-articulos-redirecciones.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11321 Pywikipediabot/1.0 - -
 application/json
 application/xml
SurakWare MediaWiki Bot/1.0 - -
 text/..
pywikipedia-wikidata3cat.py/r11181 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
 application/xml
ideasBot 1.0 Series By DGideas - -
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.39; ) - -
 text/..
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
pywikipedia-anagrama.py/r11215 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
GermCrawler - -
 application/json
pywikipediabot-wikidata_new_global.py/r11215 Pywikipediabot/1.0 - -
 application/json
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
pywikipedia-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
Wikirage.Com Statistics Bot - -
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1) - -
 application/json
pywikipedia-featured.py/r11308 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-28main.build89.py/r11131 Pywikipediabot/1.0 - -
 application/json
 application/xml
wikiscore-MakeScoreTable.py/r11152 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-vlinders_wikidata.py/r11104 Pywikipediabot/1.0 - -
 application/json
pywikipedia-iw2make2.py/r11321 Pywikipediabot/1.0 - -
 application/json
 application/xml
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
dtSearchSpider - -
 text/..
pywikipedia-wikidata_wceb.py/r11311 Pywikipediabot/1.0 - -
 application/json
 application/xml
Global-Crawler/Nutch-1.4 (Search platform crawler) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia-wikidata3cat.py/r11228 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Test Webbot - -
 text/..
WikiPlaysBot - -
 text/..
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
Wikidata-wikidata.py/r11202 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_war.py/r11311 Pywikipediabot/1.0 - -
 application/json
HTMLParser/2.0 - -
 text/..
pywikipedia-coord2.py/r11275 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-featured.py/r11324 Pywikipediabot/1.0 - -
 application/json
pywikipedia-17-progreso1911.py/r11321 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
www.integromedb.org/Crawler - -
 text/..
pywikipedia-info2.py/r11308 Pywikipediabot/1.0 - -
 application/json
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11054 Pywikipediabot/1.0 - -
 application/json
 application/xml
SchoolReviewNetworkWikiBot - -
 application/json
pywikirish-welcome.py/r11215 Pywikipediabot/1.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
pywikipedia-redirect.py/r11324 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-ko_datacheck.py/r11317 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-replace.pyc/r11215 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11265 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-ebham.py/r11321 Pywikipediabot/1.0 - -
 application/json
WikiBot/0.1 - -
 text/..
Zing-BottaBot/2.0 - -
 text/..
trunk-redirect.py/r11322 Pywikipediabot/1.0 - -
 application/json
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
TVersity Media Robot - -
 text/..
(u'python-wikitools/1.1.1 (User:Cerabot)',) - -
 application/json
wikidata_blank_items/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Wiki.java 0.27 r134 (OctraBot 2.3) - -
 text/..
www.monit24.pl-m24Bot/4.1- - -
 image/..
 text/..
pywikt-interwiki.py/r11216 (wikipedia.py) Pywikipediabot/1.0 - -
 application/xml
 application/json
Empedia Bot - -
 text/..
MediaWiki::Bot/3.005002 - -
 application/json
FAST Search Web Crawler 14.0.0325.0000 - -
 text/..
 -
bot: fr-anal - -
 application/json
usrlab-BotReversor.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-archivebot.py/r11321 Pywikipediabot/1.0 - -
 application/json
360spider-image - -
 image/..
 text/..
Bub's wikibot (Wikibot/2013031408; JWBF/1.2; Java/1.7) - -
 text/..
Twitterbot/1.0 - -
 text/..
hercule-commonscat.py/r11321 Pywikipediabot/1.0 - -
 application/json
wikidata_properties/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
notifyDisam/r11277 Pywikipediabot/2.0 - -
 application/json
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
Phantom.js bot cs-CZ,en,* -
 image/..
 text/..
pywikipedia-zzlangcat.py/r11265 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
OpenSearchServer_Bot - -
 text/..
commons-imagerecat.py/r11028 Pywikipediabot/1.0 - -
 application/json
newser-wikinewser2.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-info2.py/r11321 Pywikipediabot/1.0 - -
 application/json
bot-VM-auto-erl.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wd_move.py/r11308 Pywikipediabot/1.0 - -
 application/json
user_scripts-BotReversor.py/r11147 Pywikipediabot/1.0 - -
 application/json
A .NET Web Crawler - -
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
milog_bot/1.0 ( mail address ) - -
 text/..
ybot-ko_datastat.py/r11308 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-featured.py/r11311 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Epywikipedia-welcome.py/r11181 Pywikipediabot/1.0 - -
 application/json
trunk-redirect.py/r11308 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-welcome2.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-replace.py/r11307 (wikipedia.py) Pywikipediabot/1.0 - -
 application/xml
 application/json
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
JavaCrawler/1.1 - -
 text/..
wikbotlite/2.0 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 application/json
 image/..
 text/..
pywikipedia-17-progreso1911.py/r11308 Pywikipediabot/1.0 - -
 application/json
 application/xml
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) - -
 text/..
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
ipipan.waw.pl - nekstbot - -
 text/..
fa-welcome.py/r11054 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 image/..
 text/..
pywikipedia-ko_datacheck.py/r11308 Pywikipediabot/1.0 - -
 application/json
COIBot/1.00 - -
 text/..
Pywikipedia-anagrama.py/r11215 Pywikipediabot/1.0 - -
 application/json
admin-ipblocker-auto.py/r11281 Pywikipediabot/1.0 - -
 application/json
update-task-categories/r11182 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
Pywikipedia-new_interwiki.py/r11255 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (NewArticleAFDTagger; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-welcome.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-ebham.py/r11308 Pywikipediabot/1.0 - -
 application/json
GoogleBot - -
 text/..
ip-web-crawler.com en-us -
 text/..
pywikipedia-interwiki.py/r11322 Pywikipediabot/1.0 - -
 application/json
 application/xml
templateswdl/r11208 Pywikipediabot/2.0 - -
 application/json
pywikipedia-wikidata_irc_global.py/r11311 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_disamb_dp.py/r11200 Pywikipediabot/1.0 - -
 application/json
pywikipedia-archivebot.py/r11308 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata9nwd.py/r11181 (wikipedia.py) Pywikipediabot/1.0 - -
 application/xml
 application/json
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot en-us,en;q=0.5 -
 text/..
Py-new_interwiki.py/r11278 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
pywikipedia-bot_control.py/r504 Pywikipediabot/1.0 - -
 application/json
 application/xml
EarwigBot/0.2.dev.gitf082fca7 (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
 text/..
ybot-ko_newpage.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/r11308 Pywikipediabot/1.0 - -
 application/json
 application/xml
DeletionBot-deletion.py/r11275 Pywikipediabot/1.0 - -
 application/json
 application/xml
YBot/0.1 - -
 application/vnd.php.serialized
ybot-cat_copy.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-category_redirect.py/r11321 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.001 (windows; NT4.0; en-US; rv:1.0) Gecko/25250101 - mail address - -
 text/..
pywikipedia-wawiwtable.py/r11307 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
python-wikitools/1.2 (User:Mr.Z-bot) - -
 application/json
Metabot 0.1 - -
 text/..
CodeGator Crawler v1.0 - -
 text/..
COIBot/2.0 - -
 text/..
pywikipedia-interwiki.py/r11231 (wikipedia.py) Pywikipediabot/1.0 - -
 application/xml
 application/json
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 416 subscribers; feed-id=3852576738117026533) - -
 application/xml
 -
pywikipedia-featured.py/r11325 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzlangcat.py/r11322 Pywikipediabot/1.0 - -
 application/json
process_mlb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 text/..
theWxitBot/0.1 - -
 application/json
 image/..
Bloom.fm Meta Crawler ( mail address ) - -
 text/..
Wikibot/2.0.2 CFNetwork/609.1.4 Darwin/13.0.0 en-gb -
 image/..
 application/json
 text/..
AnomieBOT 1.0 (PUICloser; see [[User:AnomieBOT]]) - -
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) - -
 application/json
Wikibot/2.0.2 CFNetwork/609.1.4 Darwin/13.0.0 de-de -
 image/..
 application/json
 text/..
pywikipedia-klbot_wikiviajes4.py/r11311 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipediabot-wikidata_irc_global.py/r11215 Pywikipediabot/1.0 - -
 application/json
bitlybot - -
 text/..
 image/..
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
newser-wikinewser2.py/r11308 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
Wikibot/2.0.2 CFNetwork/548.1.4 Darwin/11.0.0 en-us -
 image/..
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Arkbot/0.1 alpha - -
 application/json
pywikipedia-iw2data.py/r11308 Pywikipediabot/1.0 - -
 application/json
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
AsgardBot - DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
('python-wikitools/1.1.1 (User:BernsteinBot)',) - -
 application/json
Mozilla/5.0 (compatible; Cliqusbot/0.1) en,*;q=0.5 -
 text/..
UCMore Crawler App en-us,en;q=0.5 -
 text/..
Peachy MediaWiki Bot API Version 0.1beta - -
 application/vnd.php.serialized
HTMLParser/1.6 - -
 text/..
misc-tasks-my_replace.py/r11322 Pywikipediabot/1.0 - -
 application/xml
 application/json
BuiBui-Bot/1.0 (email: mail address ) - -
 text/..
Wikibot/2.0.2 CFNetwork/609.1.4 Darwin/13.0.0 zh-cn -
 text/..
 application/json
 image/..
Zing-BottaBot/1.0 - -
 text/..
rmShortArtsTag-rmShortArtsTag/r11130 Pywikipediabot/1.0 - -
 application/json
OmnitureTestAndTargetCrawl/Nutch-1.6 (Nutch/1.6) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
My Nutch Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia-welcome2.py/r11308 Pywikipediabot/1.0 - -
 application/json
UniversalFeedParser/5.1.1 https://code.google.com/p/feedparser/ - -
 text/..
 application/xml
Pywikipedia-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
translation/r11235 Pywikipediabot/2.0 - -
 application/json
python-wikitools/1.2 (User:LaraBot) - -
 application/json
DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
Python27-pythonw.exe/r11275 Pywikipediabot/1.0 - -
 application/json
request-daemon/r11182 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
Wikibot/2.0.1 CFNetwork/609 Darwin/13.0.0 en-gb -
 image/..
 application/json
Geni ircpybot 1.0 - -
 application/json
47595.91total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Jun 17, 2013 17:32
Author:Erik Zachte (
Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers