Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 9 Jan 2013 - 30 Jan 2013 (last 12 months)
000 ⇒ k
 

 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 104,812,910 page requests (mime type text/html only!) per day are considered crawler requests, out of 605,947,230 external requests, which is 17.3%

Page requests for crawlers that specify a url in the agent string
Count
x 1000
Secondary domain
(~site) name
URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appenginetext/..Wiki.java 0.27 AppEngine-Google; (url; appid: wikipediatools) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appengineapplication/jsonMozilla/5.0 AppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appengine-Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
 code.google.com/appenginetext/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
 code.google.com/appenginetext/..Mozilla/5.0 AppEngine-Google; (url; appid: s~birthday-stats) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img) - -
 www.google.com/bot.htmlNONE/wikipedia- Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 code.google.com/appenginetext/..route-hacker alpha AppEngine-Google; (url; appid: s~routehackers) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: usawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app-cruxbot) - -
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appengineimage/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hr-pulsesubscriber) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~theunblock) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~liquid-helium) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 www.google.com/bot.htmlapplication/oggMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: your-zone) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) ja,en-us;q=0.7,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~italiatiki) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; script; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: azshaderserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-api) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azshaderserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3) - -
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~stremor-crawler) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vadim-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
facebook
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url) - -
 developers.facebook.com-facebookplatform/1.0 (url) - -
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en -
 www.bing.com/bingbot.htmapplication/vnd.php.serializedMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5 - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) en -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3 - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-GoogleBot/2.1 (url) en-us,en;q=0.5
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url) ja, *;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url) - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
msn
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) tr,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.0 (NHN Corp.; url) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/reader/faq/topic006/-Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;url;1 subscribers;) - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/-Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
 fulltext.sblog.cz/screenshot/-Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmimage/..Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmtext/..Sosospider(url) - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url) -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17 - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
 pear.php.net/application/xmlPEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - -
 173.13.143.74/bot.php-Mozilla/5.0 (compatible; YioopBot; url) - -
 173.13.143.74/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 wiki.toolserver.org/view/GeoHack-Geohack (url) -
 toolserver.org/~dispenser/-CacheThumbs/1.2 (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~dispenser/-DispensersTools (url) -
 toolserver.org/~dispenser/application/x-www-form-urlencodedDispensersTools (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/application/jsonDispensersTools (url) - -
wordpress
 josefboberg.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 tsjok45.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 kabarislam.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 jaggedykaye.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 lesliebrodie.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 beyondthelies.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 gunnyg.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 ambiya999dotcom.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.20.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggle-Huggle/2.1.19.0 url -
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API) - -
 sk.wikipedia.org/wiki/Redaktor:TeslaBot-TeslaBot (url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url) zh-cn -
finecomb
 finecomb.com/-api/1.1 (url; mail address ) - -
 finecomb.com/application/jsonapi/1.1 (url; mail address ) - -
wikidict
 www.wikidict.detext/..url - -
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url) - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
 www.majestic12.co.uk/bot.php?-Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) - -
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmlimage/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
 zookabot.comimage/..Zookabot/2.5;url - -
coccoc
 help.coccoc.vn/text/..coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.vn/-coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.vn/text/..coccoc/1.0 (url) - -
 help.coccoc.vn/text/..coccoc/1.0 (url) en-us,en;q=0.5 -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 FreeBSD 9.1-RELEASE; java 1.7.0_011; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.6.11-1-ARCH; java 1.7.0_03; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.6.11-1-ARCH; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_24; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.1; java 1.6.0_37; Asia/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.4-1-ARCH; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_07; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (torworld/any; amd64 Linux 3.4.0-5-generic; java 1.7.0_09; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; x86 Windows 7 6.1; java 1.6.0_18; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86 Windows XP 5.1; java 1.7.0_07; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-21-generic; java 1.7.0_09; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (torworld/any; amd64 Linux 3.4.0-5-generic; java 1.7.0_09; Europe/en) url en-us,en;q=0.5 -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
elcidharth
 elcidharth.comtext/..WordPress/3.6-alpha-23288; url - -
 elcidharth.comimage/..WordPress/3.6-alpha-23288; url - -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.clips.ua.ac.be/pages/patternapplication/jsonPattern/2.3 url - -
 www.tkl.iis.u-tokyo.ac.jp/~crawler/-Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) - -
 www.tkl.iis.u-tokyo.ac.jp/~crawler/image/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
cognarius
 cognarius.com-AppsArlak/1.0 (url) -
 cognarius.comapplication/jsonAppsArlak/1.0 (url) - -
 cognarius.comtext/..AppsArlak/1.0 (url) - -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url) - -
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
 goo.gl/7y4SXimage/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
creativecloudlab
 creativecloudlab.com/CclWikipediaCrawler/text/..CclWikipediaCrawler/0.3 (url; mail address ) - -
 creativecloudlab.com/CclWikipediaCrawler/text/..CclWikipediaCrawler/0.1 (url; mail address ) - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
 labs.topsy.com/butterfly/-Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
 www.gnip.com/image/..UnwindFetchor/1.0 (url) - -
 www.gnip.com/-UnwindFetchor/1.0 (url) - -
gamedipper
 www.gamedipper.comapplication/jsongamedipper.com bot (url) - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/-BSurprised WikiBox 0.1.3 (url) en
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) af -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) nl -
 bsurprised.com/application/x-www-form-urlencodedBSurprised WikiBox 0.1.3 (url) en
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robots-Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) -
trendytrack
 trendytrack.comtext/..WordPress/3.5; url - -
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url) - -
 enwp.org/User:Hellknowz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
wikiglass
 wikiglass.comtext/..url : mail address - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url) - -
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) - -
 www.mediawiki.org/-MediaWiki OAI Harvester 0.2 (url) -
 www.mediawiki.org/wiki/Extension:RSStext/..MediaWikiRSS/0.02 (url) / MediaWiki RSS extension - -
fucinamediale
 labs.fucinamediale.comtext/..Mozilla/5.0 (compatible; ExperimentalWikiBot/1.0; url) - -
plos
 alm2-dev.plos.orgapplication/jsonArticle Level Metrics - url - -
 alm.plos.orgapplication/jsonPLoS Article Level Metrics - url - -
 alm.plos.org-PLoS Article Level Metrics - url -
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url) en-us,en;q=0.5 -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20120911.190842 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
 tineye.com/crawler.htmlimage/..TinEye/1.1 (url) - -
 tineye.com/crawler.htmltext/..TinEye/1.1 (url) - -
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url) - -
 www.xbmc.orgtext/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) ja -
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
 wikimpress.org/-Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot url) - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url) - -
bibalex
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
fivemost
 www.fivemost.comapplication/jsonurl - -
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url) - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/-Drupal (url) -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
 www.rcdtokyo.com/pc2m/-Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
sf
 magpierss.sf.nettext/..MagpieRSS/0.7x (url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url) en-us,en;q=0.5 -
 liferea.sf.net/-Liferea/1.8.3 (Linux; fr_FR.UTF-8; url) -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
example
 example.com/text/..Wiki 0.0 (url; mail address ) - -
 example.com/text/..WikiExample 0.1 (url; mail address ) - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
localhost:8888
 localhost:8888image/..WordPress/3.5; url - -
 localhost:8888text/..WordPress/3.5; url - -
openindex
 www.openindex.io/en/webmasters/spider.htmltext/..Mozilla/5.0 (compatible; OpenindexSpider; url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.openindex.io/en/webmasters/spider.htmltext/..Mozilla/5.0 (compatible; OpenindexSpider; url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
n-grams
 www.n-grams.org/icorpusbot.htmltext/..iCorpusBot (url) es-es,en-us;q=0.7,en;q=0.3 -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
 www.moviecus.com/botcontactinfo.php-moviecus bot (url) -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
vermagerd
 www.vermagerd.be/wptext/..WordPress/3.4.2; url - -
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address ) - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
friendofrenia
 friendofrenia.com/text/..User-Agent: FriendoFrenia (url) - -
 friendofrenia.com/application/jsonUser-Agent: FriendoFrenia (url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url) en-us,en;q=0.5 -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
chickyrun
 chickyrun.tk/text/..ChickyBot/1.1 (url; mail address ) - -
rockmelt
 rockmelt.comtext/..RockmeltEmbedService (url) - -
linguee
 www.linguee.com/bottext/..Linguee Bot (url; mail address ) - -
 www.linguee.com/botapplication/jsonLinguee Bot (url; mail address ) - -
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url) - -
avantbrowser
 www.avantbrowser.comtext/..Advanced Browser (url) en-us,en;q=0.5 -
 www.avantbrowser.comtext/..Avant Browser (url) en-us,en;q=0.5 -
simplepie
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.95; url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 meta.stackoverflow.com/q/130398text/..Mozilla/5.0 (compatible; stackexchangebot/1.0; url) - -
dasdonkey
 www.dasdonkey.comtext/..Mozilla/5.0 (compatible; DonkeyBot/0.1; url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
 superfeedr.com-Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
webceo
 online.webceo.comtext/..Mozilla/5.0 (compatible; web-ceo-online-bot/1.0; url) - -
newsgator
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP) en-us,en;q=0.5 -
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers) en-us,en;q=0.5 -
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url) en-us,en;q=0.5 -
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url) en-us,en;q=0.5 -
medu
 medu.pl/bottext/..Mozilla/5.0 (compatible; Medubot/1.0; url) - -
 medu.pl/bot-Mozilla/5.0 (compatible; Medubot/1.0; url) -
site-shot
 www.site-shot.com/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -
 www.site-shot.com/text/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -
feedshow
 www.feedshow.comtext/..FeedshowOnline (url) en-us,en;q=0.5 -
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber) en-us,en;q=0.5 -
seomoz
 www.seomoz.org/dp/rogerbottext/..rogerbot/1.0 (url, mail address ) - -
 www.seomoz.org/dp/rogerbottext/..rogerbot/1.0 (url, mail address ) - -
igrec
 www.igrec.ca/projectstext/..Wikitionary Text Parser 0.2 (url) - -
owasp
 www.owasp.org/index.php/Category:OWASP_DirBuster_Projecttext/..DirBuster-0.12 (url) - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Bee-Python-urllib/2.7 (WikiApiary; Bumble Bee; url) -
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonPython-urllib/2.7 (WikiApiary; Bumble Bee; url) - -
theworldtopbrands
 theworldtopbrands.comtext/..WordPress/3.4.2; url - -
metamagazine
 metamagazine.comtext/..WordPress/3.4.2; url - -
 metamagazine.com-WordPress/3.4.2; url -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
localhost
 localhosttext/..Mozilla/5.0 (compatible; heritrix/2.0.2 url) - -
 localhost/wordpresstext/..WordPress/3.5; url - -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.1 (url) - -
 www.yougorhymes.com/site/rhyme-bot-RhymeBot/0.1 (url) -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
onet
wikimedia
 tools.wikimedia.de/~para/GeoCommons/text/..url - -
 tools.wikimedia.de/~para/GeoCommons/-url -
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.4 (url) - -
 www.backgroundswitcher.com/image/..John's Background Switcher 4.6 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.3 (url) - -
nationallibrary
 www.nationallibrary.fi/text/..Mozilla/5.0 (compatible; heritrix/1.14.0url) - -
 www.nationallibrary.fi/-Mozilla/5.0 (compatible; heritrix/1.14.0url) - -
 www.nationallibrary.fi/image/..Mozilla/5.0 (compatible; heritrix/1.14.0url) - -
pinterest
 pinterest.com/image/..Pinterest/0.1 url - -
 pinterest.com/text/..Pinterest/0.1 url - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
gsd-software
 www.gsd-software.comtext/..GSDCrawler (url) - -
blisshq
 www.blisshq.comimage/..bliss/20121128 url - -
 www.blisshq.comtext/..bliss/20121128 url - -
wotbox
 www.wotbox.com/bot/text/..Wotbox/2.01 (url) - -
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url) en-us -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckpreview.htmltext/..DuckDuckPreview/1.0; (url) - -
 duckduckgo.com/duckduckpreview.html-DuckDuckPreview/1.0; (url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9 en-us,en;q=0.5 -
weborking
 weborking.comtext/..Weborking(url) - -
 weborking.comapplication/jsonWeborking(url) - -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl-ja.htmltext/..ICC-Crawler (Mozilla-compatible; mail address ; url) ja -
watchmouse
esciudad
 www.esciudad.com/application/jsonEsciudad/1.0 (url) - -
femfamilyinternational
 femfamilyinternational.org/faithtext/..WordPress/3.5; url - -
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -
heartrails
 capture.heartrails.com/image/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.17 ja,en-us;q=0.7,en;q=0.3 -
 capture.heartrails.com/text/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.17 ja,en-us;q=0.7,en;q=0.3 -
toshiba
 www.toshiba.co.jp/rdc/about/crawl_info_en.htmtext/..TosCrawler/Nutch-1.4 (url; ' mail address dot co dot jp') ja,ja-jp;q=0.7,*;q=0.3 -
compspy
 www.compspy.com/spider.htmltext/..Mozilla/5.0 (compatible; CompSpyBot/1.0; url) - -
134211.550000016total

Page requests for probable crawlers, recognized by keyword
Count
x 1000
Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0 - -
 application/json
 application/xml
 text/..
 -
 application/x-www-form-urlencoded
 image/..
 application/pdf
PythonWikipediaBot/1.0 -
 -
 application/x-www-form-urlencoded
spider - -
 text/..
 application/vnd.php.serialized
 application/yaml
 application/json
 image/..
 -
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 -
 text/..
 image/..
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
LinkParser/2.0 - -
 text/..
DotNetWikiBot/2.101 (Unix 2.6.32.39; ) - -
 text/..
 application/xml
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
Peachy MediaWiki Bot API Version 1.0 -
 -
wikiwix-bot-3.0 - -
 text/..
 -
tigerbot - -
 application/json
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
 application/pdf
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
 text/..
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
 application/rsd+xml
ClueBot/1.1 - -
 application/vnd.php.serialized
 text/..
Answersbot - -
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider - -
 text/..
 application/xml
 application/json
 application/vnd.php.serialized
ClueBot/2.0 - -
 application/vnd.php.serialized
www.integromedb.org/Crawler - -
 text/..
 -
 application/xml
TrueKnowledgeBot bot mail address > - -
 application/xml
 application/vnd.php.serialized
 image/..
Pywikipediabot/2.0 - -
 application/json
 text/..
gsa-crawler (Enterprise; T3-P9JWVCTT9WWGY; mail address ) - -
 text/..
Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
Wikipath Bot (email: mail address ) - -
 application/json
Mozilla 5.0 (Apibot 0.32) - -
 application/vnd.php.serialized
 text/..
crawl/0.4 libcrawl/0.3 - -
 text/..
 image/..
 application/json
 application/ogg
DigitalsmithsBot - -
 text/..
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 -
 image/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
 image/..
 application/ogg
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 -
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
MediaWiki::Bot/3.2.6 - -
 application/json
 -
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
AarghBot Linux - -
 text/..
Tawbot (public svn release; plwiki) - -
 text/..
WikiCatResearchBot ( mail address ) - -
 text/..
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
 -
mail address - -
 application/vnd.php.serialized
 text/..
HTMLParser/1.6 - -
 text/..
 -
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
 image/..
 application/ogg
www.monit24.pl-m24Bot/4.0- - -
 -
 image/..
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
ExactusBot-v0.1 - -
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.35; ) - -
 text/..
Hulubot - -
 text/..
gsa-crawler (Enterprise; T3-F5C5JE7XKWWBK; mail address ) - -
 text/..
 -
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 -
SchoolReviewNetworkWikiBot - -
 application/json
 text/..
XBot v1.0 using MER-C's Wiki.java - -
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
SearchBot - -
 text/..
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
dtSearchSpider - -
 text/..
php wikibot classes -
 -
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
YBot/0.1 - -
 application/vnd.php.serialized
GermCrawler - -
 application/json
 text/..
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
Test Webbot - -
 text/..
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; ) - -
 text/..
 application/xml
- Peachy MediaWiki Bot API Version 1.0 -
 multipart/form-data
mySpider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
Phantom.js bot cs-CZ,en,* -
 image/..
 text/..
 -
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
MediaWiki::Bot/3.005002 - -
 application/json
360spider-image - -
 image/..
 text/..
milog_bot/1.0 ( mail address ) - -
 text/..
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
 application/x-www-form-urlencoded
Spinuf Spider - -
 text/..
 -
Twitterbot/1.0 - -
 text/..
 image/..
 -
 application/pdf
JavaCrawler/1.1 - -
 text/..
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
SurakWare MediaWiki Bot/1.0 - -
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
- PythonWikipediaBot/1.0 -
 multipart/form-data
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
crawler - -
 image/..
 text/..
wikiwix-bot-3.0 en -
 text/..
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
GoogleBot-Image/1.0 -
 -
 image/..
OrlodrimBot/1.0 - -
 text/..
 -
 application/x-www-form-urlencoded
HRoestBot, de-wikipedia using pywikipedia framework -
 application/x-www-form-urlencoded
 -
WPBot 1.0 - -
 text/..
 image/..
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 image/..
 text/..
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) - -
 text/..
XLinkBot/1.00 - -
 text/..
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
Junut Bot 1.0.3 ru-RU,en,* -
 text/..
 -
 application/xml
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) -
 -
Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl - -
 text/..
 -
HTMLParser/2.0 - -
 text/..
 -
 application/xml
WikiBot/0.1 - -
 text/..
 image/..
TicketsBot/0.1 - -
 text/..
FAST Search Web Crawler 14.0.0325.0000 -
 -
 text/..
FAST Search Web Crawler 14.0.0325.0000 - -
 text/..
 -
 image/..
 application/ogg
DotNetWikiBot/2.101 (Unix 2.6.32.45; ) - -
 text/..
DotNetWikiBot/2.101 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
COIBot/1.00 - -
 text/..
MBBot/1.0.0 en-us,en;q=0.5 -
 text/..
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
 image/..
Zing-BottaBot/2.0 - -
 text/..
TVersity Media Robot - -
 text/..
Empedia Bot - -
 text/..
wikbotlite/2.0 CFNetwork/609 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
UCMore Crawler App en-us,en;q=0.5 -
 text/..
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot en-us,en;q=0.5 -
 text/..
theWxitBot/0.1 - -
 application/json
 image/..
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 text/..
Metabot 0.1 - -
 text/..
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 416 subscribers; feed-id=3852576738117026533) - -
 application/xml
 -
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
HersfoldIRCBot version 1.2.1 - -
 text/..
 -
Geni ircpybot 1.0 - -
 application/json
 text/..
 application/xml
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
COIBot/2.0 - -
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 -
Wikibot/2.0.2 CFNetwork/548.1.4 Darwin/11.0.0 en-us -
 image/..
 application/json
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
python-wikitools/1.2 (User:BernsteinBot) -
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; NewThingForMeBot; mailto: mail address ) - -
 text/..
 image/..
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
python-wikitools/1.2 (User:BernsteinBot) - -
 application/json
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
IsraBot - -
 text/..
MerlBot -
 -
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
Geni ircpybot 1.0 -
 -
LWNutch/Nutch-1.4 (another scientific bot - we accept your robots.txt! ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-gb -
 image/..
 application/json
 text/..
 -
LauschenBot/1.1 ( mail address ) - -
 text/..
python-wikitools/1.2 (User:LaraBot) - -
 application/json
LauschenBot/1.1 ( mail address ) -
 -
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 de-de -
 image/..
 application/json
 text/..
 -
FAST Enterprise Crawler 6 used by contosoa.com ( mail address ) - -
 text/..
MerlBot - -
 application/vnd.php.serialized
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
python-wikitools/1.2 (User:LaraBot) -
 application/x-www-form-urlencoded
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
IsraBot -
 -
NoyaBot - -
 text/..
Wikibot/2.0.1 CFNetwork/609 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
Robots.txt finder - -
 text/..
GoogleBot - -
 text/..
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 zh-cn -
 image/..
 application/json
 text/..
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
OpenLink Virtuoso RDF crawler - -
 image/..
 text/..
 -
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.2/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
bitlybot - -
 text/..
 image/..
 -
MediaWiki::PerlBot/User:Wdwdbot - -
 application/json
bot: fr-anal - -
 application/json
MediaWiki::Bot 3.1.5 -
 application/x-www-form-urlencoded
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
MediaWiki::Bot 3.1.5 - -
 application/json
SoxBot PHP -
 -
Inlibris.com XMLBot/1.0 - -
 text/..
EarwigBot/0.2.dev.git4ff7612a (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) -
 application/x-www-form-urlencoded
DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
39687.77total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Tue, Dec 24, 2013 13:08
Author:Erik Zachte (
Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers