Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Feb 2013 - 28 Feb 2013 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 113,552,110 page requests (mime type text/html only!) per day are considered crawler requests, out of 633,864,640 external requests, which is 17.9%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..Wiki.java 0.27 AppEngine-Google; (url; appid: wikipediatools) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 www.google.com/bot.htmlNONE/wikipedia- Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/text/..MyBot/1.0 (url) - -
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikigameapp) - -
 code.google.com/appengineapplication/jsonMozilla/5.0 AppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; script; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/p/gwtwiki/text/..JavaWikipediaAPI/3.0 url - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: usawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cyber-zone) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy0) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: your-zone) - -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~italiatiki) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~stremor-crawler) - -
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app-cruxbot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9) - -
facebook
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url) - -
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmapplication/vnd.php.serializedMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5 - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3 - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: appcodes) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 google.com/text/..GoogleBot (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-GoogleBot/2.1 (url) en-us,en;q=0.5
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.google.com/bot.html-GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexImages/3.0; url) -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) en-us,en;q=0.5 -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/application/jsonYeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) tr,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) -
 search.msn.com/msnbot.htmimage/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url)
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/-Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
 fulltext.sblog.cz/screenshot/-Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url) -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; ) PHP/5.4 - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17 - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
 pear.php.net/application/xmlPEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - -
 173.13.143.74/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url) - -
 173.13.143.74/bot.php-Mozilla/5.0 (compatible; YioopBot; url) - -
wordpress
 josefboberg.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 thesandymonocle.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 sreaves32.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 jaggedykaye.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 lesliebrodie.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 1rgcruz.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 cuthulan.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 tsjok45.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 halokue.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 imagenssagradas.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 thaiintelligentnews.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 barbielistholland.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 gunnyg.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 onanatomia.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 wordsdohurtus.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
 investigationsoanisetoceanographiee.wordpress.comtext/..WordPress/3.6-alpha-23334; url - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 wiki.toolserver.org/view/GeoHack-Geohack (url) -
 toolserver.org/~dispenser/-CacheThumbs/1.2 (url) -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~dispenser/-DispensersTools (url) -
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/application/x-www-form-urlencodedDispensersTools (url) -
 toolserver.org/~dispenser/application/jsonDispensersTools (url) - -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.20.0 url - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation-WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url - -
 sk.wikipedia.org/wiki/Redaktor:TeslaBot-TeslaBot (url) - -
 sk.wikipedia.org/wiki/Redaktor:TeslaBotapplication/vnd.php.serializedTeslaBot (url) - -
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API) - -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
 zookabot.comimage/..Zookabot/2.5;url - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) - -
 shoulu.jike.com/spider.htmlapplication/jsonMozilla/5.0 (compatible; JikeSpider; url) - -
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -
 www.majestic12.co.uk/bot.php?-Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) it -
coccoc
 help.coccoc.com/text/..coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/-coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.vn/text/..coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.vn/-coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..coccoc/1.0 (url) en-us,en;q=0.5 -
 help.coccoc.com/text/..coccoc/1.0 (url) - -
 help.coccoc.com/image/..coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.vn/text/..coccoc/1.0 (url) en-us,en;q=0.5 -
cognarius
 cognarius.com-AppsArlak/1.0 (url) -
 cognarius.comapplication/jsonAppsArlak/1.0 (url) - -
 cognarius.comtext/..AppsArlak/1.0 (url) - -
scrapy
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
localhost
 localhost/wordpresstext/..WordPress/3.5.1; url - -
 localhost/wp2text/..WordPress/3.5; url - -
 localhost/uktext/..WordPress/3.5.1; url - -
 localhost/wptext/..WordPress/3.5; url - -
wikidict
 www.wikidict.detext/..url - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url) - -
plos
 alm.plos.org-PLoS Article Level Metrics - url -
 alm.plos.orgapplication/jsonPLoS Article Level Metrics - url - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
 labs.topsy.com/butterfly/-Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_04; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 3.6.10-nrj-desktop-1rosa; java 1.7.0_b147-icedtea; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-34-generic; java 1.6.0_27; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-rt-amd64; java 1.6.0_24; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.26-2-686; java 1.6.0_18; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.2; java 1.6.0_37; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-999-generic; java 1.7.0_15; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-34-generic; java 1.6.0_27; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-23-generic; java 1.6.0_24; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-4-rt-amd64; java 1.6.0_24; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.5.0-23-generic; java 1.6.0_24; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_15; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (webportal-global; amd64 Linux 3.6.10-nrj-desktop-1rosa; java 1.7.0_b147-icedtea; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_24; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; x86_64 Mac OS X 10.8.2; java 1.6.0_37; Europe/de) url en-us,en;q=0.5 -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
 goo.gl/7y4SXimage/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
 www.archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
linguee
 www.linguee.com/bottext/..Linguee Bot (url; mail address ) - -
 www.linguee.com/bot-Linguee Bot (url; mail address ) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
 www.gnip.com/image/..UnwindFetchor/1.0 (url) - -
elcidharth
 elcidharth.comtext/..WordPress/3.6-alpha-23334; url - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/-BSurprised WikiBox 0.1.3 (url) en
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) af -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.clips.ua.ac.be/pages/patternapplication/jsonPattern/2.3 url - -
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
 www.clips.ua.ac.be/pages/patterntext/..Pattern/2.3 url - -
 www.tkl.iis.u-tokyo.ac.jp/~crawler/-Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url) - -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url) en-us,en;q=0.5 -
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
friendofrenia
 friendofrenia.com/text/..User-Agent: FriendoFrenia (url) - -
 friendofrenia.com/application/jsonUser-Agent: FriendoFrenia (url) - -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url) - -
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
 wikimpress.org/-Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url) - -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
 tineye.com/crawler.htmlimage/..TinEye/1.1 (url) - -
drupal
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/-Drupal (url) -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
bibalex
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
chickyrun
 chickyrun.tk/text/..ChickyBot/1.1 (url; mail address ) - -
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot url) - -
eurde
 handbagshoesman.eurde.infotext/..WordPress/3.3.1; url - -
 dolls-hobby.eurde.infotext/..WordPress/3.3.1; url - -
 watchmanwoman.eurde.infotext/..WordPress/3.3.1; url - -
 audio-gps.eurde.infotext/..WordPress/3.3.1; url - -
 applian.eurde.infotext/..WordPress/3.3.1; url - -
 tv.eurde.infotext/..WordPress/3.3.1; url - -
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/cgiffard/node-simplecrawler.gittext/..Node/simplecrawler 0.1.6 (url) - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
wikiglass
 wikiglass.comtext/..url : mail address - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
theworldtopbrands
 theworldtopbrands.comtext/..WordPress/3.4.2; url - -
domain
 yourwebsite.domain.com/text/..Mozilla/5.0 (compatible; heritrix/1.8.0 url) - -
 yourwebsite.domain.com/image/..Mozilla/5.0 (compatible; heritrix/1.8.0 url) - -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url) - -
sf
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url) en-us,en;q=0.5 -
 magpierss.sf.nettext/..MagpieRSS/0.7x (url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url) en-us,en;q=0.5 -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 meta.stackoverflow.com/q/130398text/..Mozilla/5.0 (compatible; stackexchangebot/1.0; url) - -
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url) en-us,en;q=0.5 -
uni-potsdam
 www.hpi.uni-potsdam.de/meinel/forschung/web_30/blog_intelligence.htmltext/..HPI-BI-Crawler/0.1(url)/Nutch-2.0-dev - -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
 www.rcdtokyo.com/pc2m/-Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.1 (url) - -
 www.yougorhymes.com/site/rhyme-bot-RhymeBot/0.1 (url) -
example
 example.com/MyCoolTool/text/..User-Agent: MyCoolTool/1.1 (url; mail address ) - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X101.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
n-grams
 www.n-grams.org/icorpusbot.htmltext/..iCorpusBot (url) es-es,en-us;q=0.7,en;q=0.3 -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
hagjed
 hagjed.com/boomustext/..WordPress/3.5.1; url - -
 hagjed.com/boomus2text/..WordPress/3.5.1; url - -
 hagjed.com/1text/..WordPress/3.3.1; url - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
rockmelt
 rockmelt.comtext/..RockmeltEmbedder (url; mail address ) - -
 rockmelt.comtext/..RockmeltEmbedService (url) - -
vermagerd
 www.vermagerd.be/wptext/..WordPress/3.4.2; url - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
 www.moviecus.com/botcontactinfo.php-moviecus bot (url) -
antoniusyudi
 antoniusyudi.com/grab/UK/uk8text/..WordPress/3.5.1; url - -
 antoniusyudi.com/grab/UK/uk9text/..WordPress/3.5.1; url - -
 antoniusyudi.com/grab/UK/uk1text/..WordPress/3.5.1; url - -
 antoniusyudi.com/grab/UK/uk4text/..WordPress/3.5.1; url - -
 antoniusyudi.com/grab/UK/uk3text/..WordPress/3.5.1; url - -
 antoniusyudi.com/grab/UK/uk5text/..WordPress/3.5.1; url - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) - -
 www.mediawiki.org/-MediaWiki OAI Harvester 0.2 (url) -
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental) - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
weitz
 weitz.de/drakma/text/..Drakma/1.2.9 (SBCL 1.1.2; Linux; 2.6.32-042stab072.10; url) - -
 weitz.de/drakma/text/..Drakma/1.2.9 (SBCL 1.1.4; Linux; 3.1.0-1.2-xen; url) - -
avantbrowser
 www.avantbrowser.comtext/..Advanced Browser (url) en-us,en;q=0.5 -
 www.avantbrowser.comtext/..Avant Browser (url) en-us,en;q=0.5 -
budifratz
 budifratz.com/fr06text/..WordPress/3.5.1; url - -
 budifratz.com/fr09text/..WordPress/3.5.1; url - -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
dropcolors
 amzn.dropcolors.com/ardi5text/..WordPress/3.3.1; url - -
chiavicodificate
 www.chiavicodificate.ittext/..WordPress/3.1.1; url - -
newsgator
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers) en-us,en;q=0.5 -
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP) en-us,en;q=0.5 -
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url) en-us,en;q=0.5 -
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url) en-us,en;q=0.5 -
toshiba
 www.toshiba.co.jp/rdc/about/crawl_info_en.htmtext/..TosCrawler/Nutch-1.6 (url; ' mail address dot co dot jp') en-us,en-gb,en;q=0.99,*;q=0.01 -
 www.toshiba.co.jp/rdc/about/crawl_info.htmtext/..TosCrawler/Nutch-1.6 (url; ' mail address dot co dot jp') en-us,en-gb,en;q=0.99,*;q=0.01 -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
feedshow
 www.feedshow.comtext/..FeedshowOnline (url) en-us,en;q=0.5 -
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber) en-us,en;q=0.5 -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
alex-v
 alex-v.comtext/..WordPress/3.5.1; url - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
fivemost
 www.fivemost.comapplication/jsonurl - -
zapbot
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url) - -
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url) - -
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url) - -
forecheck
 www.forecheck.comtext/..Forecheck/0.5.0 (url) - -
starsandcatz
 www.starsandcatz.com.auapplication/jsonWikiBox 2.0p-betaTestVersion (url) en-US -
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url) en-us -
weborking
 weborking.comtext/..Weborking(url) - -
 weborking.comapplication/jsonWeborking(url) - -
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address ) - -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.6 (url) - -
 www.backgroundswitcher.com/image/..John's Background Switcher 4.4 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.4 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.3 (url) - -
openindex
 www.openindex.io/en/webmasters/spider.htmltext/..Mozilla/5.0 (compatible; OpenindexSpider; url) en-us,en-gb,en;q=0.7,*;q=0.3 -
hansgroheersatzteile
 hansgroheersatzteile.comtext/..WordPress/3.5; url - -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
watchmouse
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9 en-us,en;q=0.5 -
finecomb
 finecomb.com/application/jsonapi/1.1 (url; mail address ) - -
 finecomb.com/-api/1.1 (url; mail address ) - -
site-shot
 www.site-shot.com/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -
 www.site-shot.com/text/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
site
 www.site.com/application/jsonBot/2.0 (url) - -
 www.site.com/-Bot/2.0 (url) -
metamagazine
 metamagazine.comtext/..WordPress/3.4.2; url - -
 metamagazine.com-WordPress/3.4.2; url -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -
seomoz
 www.seomoz.org/dp/rogerbottext/..rogerbot/1.0 (url, mail address ) - -
sourceforge
 linkchecker.sourceforge.net/text/..Mozilla/5.0 (compatible; LinkChecker/8.2; url) - -
 linkchecker.sourceforge.net/text/..LinkChecker/5.1 (url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
mijntweedescherm
 mijntweedescherm.nltext/..MijnTweedeScherm/0.1 (url; mail address ) BasedOnSuperLib/1.4 - -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl-ja.htmltext/..ICC-Crawler (Mozilla-compatible; mail address ; url) ja -
medu
 medu.pl/bottext/..Mozilla/5.0 (compatible; Medubot/1.0; url) - -
 medu.pl/bot-Mozilla/5.0 (compatible; Medubot/1.0; url) -
kalooga
 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
142225.070000006total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0 - -
 application/json
 application/xml
 text/..
 -
 application/x-www-form-urlencoded
 image/..
 application/pdf
spider - -
 text/..
 application/vnd.php.serialized
 application/json
 image/..
 -
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
 application/json
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 -
 text/..
 image/..
PythonWikipediaBot/1.0 -
 -
 application/x-www-form-urlencoded
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
pywikipedia-newar4ur.py/r11029 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-newar4ur.py/r11029 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
LinkParser/2.0 - -
 text/..
DotNetWikiBot/2.101 (Unix 2.6.32.45; ) - -
 text/..
 application/xml
tigerbot - -
 -
 application/json
 text/..
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
 text/..
Peachy MediaWiki Bot API Version 1.0 -
 -
wikiwix-bot-3.0 - -
 text/..
 -
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
ClueBot/1.1 - -
 application/vnd.php.serialized
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
 text/..
ClueBot/2.0 - -
 application/vnd.php.serialized
GoogleBot - -
 text/..
 image/..
 application/ogg
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
yasbot-hamsang.py/r11054 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
yasbot-hamsang.py/r11054 Pywikipediabot/1.0 - -
 application/json
 application/xml
TrueKnowledgeBot bot mail address > - -
 application/xml
 application/vnd.php.serialized
 image/..
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
MediaWiki::Bot/3.3.1 - -
 application/json
 text/..
crawl/0.4 libcrawl/0.3 - -
 text/..
 image/..
pywikipedia-radeh7.py/r11027 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-radeh7.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla 5.0 (Apibot 0.32) - -
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 -
 image/..
 video/webm
Peachy MediaWiki Bot API Version 2.0 (beta) - -
 application/vnd.php.serialized
 text/..
DigitalsmithsBot - -
 text/..
Answersbot - -
 text/..
www.integromedb.org/Crawler - -
 text/..
 -
 application/xml
 image/..
fa-welcome.py/r11054 Pywikipediabot/1.0 - -
 application/json
fa-welcome.py/r11054 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
 -
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 -
MediaWiki::Bot/3.2.6 - -
 application/json
 text/..
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
 application/json
Peachy MediaWiki Bot API Version 2.0 (beta) -
 -
pywikipedia-redirect.py/r11071 Pywikipediabot/1.0 - -
 application/json
Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
 image/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
Tawbot (public svn release; plwiki) - -
 text/..
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
rewrite-maj_articles_recents.py/r11028 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r11071 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
rewrite-maj_articles_recents.py/r11028 Pywikipediabot/2.0 - -
 application/json
pywikipedia-interwiki.py/r11096 Pywikipediabot/1.0 - -
 application/xml
 application/json
Wikibot/2.0.2 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
pywikipedia-interwiki.py/r11080 Pywikipediabot/1.0 - -
 application/json
 application/xml
WikiCatResearchBot ( mail address ) - -
 text/..
AdMedia bot - -
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
 image/..
 application/ogg
wikidata-delsitelinks.py/r11060 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipediabot-wikidata.py/r11076 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11027 Pywikipediabot/1.0 - -
 application/json
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 -
 -
pywikipedia-radeh.py/r11071 Pywikipediabot/1.0 -
 -
pywikipedia-radeh.py/r11071 Pywikipediabot/1.0 - -
 application/json
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-redirect.py/r11027 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - -
 application/json
Junut Bot 1.0.3 ru-RU,en,* -
 text/..
 -
 application/xml
 application/x-external-editor
pywikipedia-interwiki.py/r11085 Pywikipediabot/1.0 - -
 application/json
 application/xml
Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
JohnFLBot/1.1 - -
 application/vnd.php.serialized
pywikipedia-interwiki.py/r11028 Pywikipediabot/1.0 - -
 application/json
 application/xml
 -
 application/x-www-form-urlencoded
pywikipedia-iwiki2_TS.py/r11080 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-dave_sezioni.py/r11060 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-interwiki.py/r11054 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-iwiki2_TS.py/r11029 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11102 Pywikipediabot/1.0 - -
 application/json
 application/xml
 application/x-www-form-urlencoded
 -
pywikipedia-iwiki2_TS.py/r11029 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
 image/..
 application/ogg
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
www.monit24.pl-m24Bot/4.0- - -
 -
 image/..
 text/..
pywikipedia-iwiki2_TS.py/r11080 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-interwiki.py/r11103 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
SchoolReviewNetworkWikiBot - -
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 -
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
pywikipedia-redirect.py/r11080 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11085 Pywikipediabot/1.0 - -
 application/json
 text/..
GermCrawler - -
 application/json
pywikipedia-interwiki.py/r11071 Pywikipediabot/1.0 - -
 application/json
 application/xml
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
enwiki_removal/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
- Peachy MediaWiki Bot API Version 2.0 (beta) -
 multipart/form-data
DotNetWikiBot/2.100 (Unix 3.2.0.37; ) - -
 text/..
php wikibot classes -
 -
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
pywikipedia-redirect.py/r11080 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-interwiki.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywiki-deledpimage.py/r11029 Pywikipediabot/1.0 -
 -
mail address - -
 application/vnd.php.serialized
 text/..
enwiki_removal/r-1 (unknown) Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
flagged revision bot. - -
 application/json
 text/..
pywiki-deledpimage.py/r11029 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11085 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-hamsang.py/r11073 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
wikidata_properties/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-hamsang.py/r11073 Pywikipediabot/1.0 - -
 application/json
pywiki-deledpimage.py/r11027 Pywikipediabot/1.0 -
 -
pywiki-deledpimage.py/r11027 Pywikipediabot/1.0 - -
 application/json
OrlodrimBot/1.0 - -
 text/..
 -
 application/x-www-form-urlencoded
pywiki-deledpimage.py/r11071 Pywikipediabot/1.0 -
 -
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
pywiki-deledpimage.py/r11071 Pywikipediabot/1.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
pywikipedia-commonscat.py/r11054 Pywikipediabot/1.0 - -
 application/json
 application/xml
FAST Search Web Crawler 14.0.0325.0000 - -
 text/..
 -
 image/..
 application/ogg
Phantom.js bot cs-CZ,en,* -
 image/..
 text/..
pywikipedia-radeh.py/r11029 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikipedia-radeh.py/r11029 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider - -
 text/..
 application/xml
 application/json
fa-hamsang.py/r11054 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-radeh.py/r11060 Pywikipediabot/1.0 -
 -
pywikipedia-interwiki.py/r11073 Pywikipediabot/1.0 - -
 application/xml
 application/json
 -
 application/x-www-form-urlencoded
fa-hamsang.py/r11054 Pywikipediabot/1.0 - -
 application/json
 application/xml
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
 -
pywikipedia-zzredirectyeh.py/r11088 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzredirectyeh.py/r11088 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r11029 Pywikipediabot/1.0 - -
 application/json
MorbZ-Bot - -
 application/json
pywikipedia-redirect.py/r11102 Pywikipediabot/1.0 - -
 application/json
Test Webbot - -
 text/..
pywikipedia-interwiki.py/r11131 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-radeh.py/r11060 Pywikipediabot/1.0 - -
 application/json
- Peachy MediaWiki Bot API Version 1.0 -
 multipart/form-data
pywikipedia-redirect.py/r11096 Pywikipediabot/1.0 - -
 application/json
Wikirage.Com Statistics Bot - -
 text/..
pywikipedia-redirect.py/r11131 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 -
 -
 image/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
pywikipedia-interwiki.py/r11029 Pywikipediabot/1.0 - -
 application/xml
 application/json
pywikipedia-redirect.py/r11076 Pywikipediabot/1.0 - -
 application/json
UAA-UAA.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11054 Pywikipediabot/1.0 - -
 application/json
wikiwix-bot-3.0 en -
 text/..
pywikipedia-anagrama.py/r10663 Pywikipediabot/1.0 - -
 application/json
FAST Search Web Crawler 14.0.0325.0000 -
 -
wikidata-delsitelinks.py/r11060 Pywikipediabot/1.0 -
 -
trunk-checkimages.py/r11072 Pywikipediabot/1.0 - -
 application/json
dtSearchSpider - -
 text/..
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
 application/x-www-form-urlencoded
Twitterbot/1.0 - -
 text/..
 image/..
 -
 application/pdf
pywikipedia-interwiki.py/r11076 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11136 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
SurakWare MediaWiki Bot/1.0 - -
 text/..
 application/xml
TVersity Media Robot - -
 text/..
pywikipedia-redirect.py/r11136 Pywikipediabot/1.0 - -
 application/json
pywiki-deledpimage.py/r11060 Pywikipediabot/1.0 -
 -
MediaWiki::Bot/3.005002 - -
 application/json
pywiki-deledpimage.py/r11060 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11029 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
TjBot-1-interwiki.py/r11028 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11131 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-redirect.py/r11130 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11110 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11096 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
pywikipedia-redirect.py/r11104 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-radeh.py/r11027 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/r11113 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-radeh.py/r11027 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
milog_bot/1.0 ( mail address ) - -
 text/..
pywikipedia-redirect.py/r11130 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
WikiBot/0.1 - -
 text/..
 image/..
trunk-checkimages.py/r11085 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11086 Pywikipediabot/1.0 - -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1) -
 application/x-www-form-urlencoded
wikidata_properties/r-1 (unknown) Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1) - -
 application/json
pynik-checkimages.py/r11071 Pywikipediabot/1.0 - -
 application/json
crawler - -
 image/..
 text/..
pywikipedia-redirect.py/r11060 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-redirect.py/r11103 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-redirect.py/r11136 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-interwiki.py/r11105 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11086 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-basicreferenceslist.py/r11070 (wikipedia.py) Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r11102 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywiki-deledpimage.py/r11054 Pywikipediabot/1.0 -
 -
pywikipedia-redirect.py/r11110 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-redirect.py/r11103 Pywikipediabot/1.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
pywikipedia-redirect.py/r11060 Pywikipediabot/1.0 - -
 application/json
ybot-ko_datastat.py/r11114 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pynik-checkimages.py/r11071 Pywikipediabot/1.0 -
 -
pywikipedia-interwiki.py/r11072 Pywikipediabot/1.0 - -
 application/json
 application/xml
 application/x-www-form-urlencoded
 -
pywikipedia-redirect.py/r11054 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-basicreferenceslist.py/r11070 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
JavaCrawler/1.1 - -
 text/..
 image/..
 application/ogg
pynik-checkimages.py/r11027 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r11076 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywiki-deledpimage.py/r11054 Pywikipediabot/1.0 - -
 application/json
pywikipedia3-interwiki.py/r11080 Pywikipediabot/1.0 - -
 application/xml
 application/json
ybot-ko_datastat.py/r11114 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-iwiki2_TS.py/r11076 Pywikipediabot/1.0 - -
 application/json
 application/xml
pynik-checkimages.py/r11027 Pywikipediabot/1.0 - -
 application/json
pywikipedia-hamsang.py/r11080 Pywikipediabot/1.0 -
 -
pywikipedia-redirect.py/r11104 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
HTMLParser/2.0 - -
 text/..
 -
pywikipedia-interwiki.py/r11130 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-hamsang.py/r11080 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/r11099 Pywikipediabot/1.0 - -
 application/json
 application/xml
HRoestBot, de-wikipedia using pywikipedia framework -
 application/x-www-form-urlencoded
 -
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) -
 -
TjBot-1-interwiki.py/r11099 Pywikipediabot/1.0 - -
 application/json
 application/xml
trunk-checkimages.py/r11111 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 image/..
 text/..
pywikipedia-interwiki.py/r11102 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
nn-interwiki.py/r11076 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) - -
 text/..
nn-interwiki.py/r11076 Pywikipediabot/1.0 - -
 application/json
 application/xml
('python-wikitools/1.1.1 (User:BernsteinBot)',) - -
 application/json
pywikipedia-iwiki2_TS.py/r11076 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikipedia-newar4ur.py/r11135 Pywikipediabot/1.0 - -
 application/json
HTMLParser/1.6 - -
 text/..
pywikipedia-zzfatranslator.py/r11071 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
('python-wikitools/1.1.1 (User:BernsteinBot)',) -
 application/x-www-form-urlencoded
SD Crawler/Nutch-1.5.1 (automated Crawler) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
COIBot/1.00 - -
 text/..
svnpywikipedia-interwiki.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
TicketsBot/0.1 - -
 text/..
 image/..
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
pywikipedia-interwiki.py/r11027 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
360spider-image - -
 image/..
 text/..
Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl - -
 text/..
trunk-interwiki.py/r11103 Pywikipediabot/1.0 - -
 application/json
 application/xml
PhronesisSpider - -
 text/..
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 text/..
pynik-checkimages.py/r11029 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-interwiki.py/r11088 Pywikipediabot/1.0 - -
 application/json
 application/xml
pynik-checkimages.py/r11029 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
trunk-interwiki.py/r11081 Pywikipediabot/1.0 - -
 application/json
 application/xml
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-zzfatranslator.py/r11071 Pywikipediabot/1.0 - -
 application/json
pywikipedia2-interwiki.py/r11071 Pywikipediabot/1.0 - -
 application/json
 application/xml
Zing-BottaBot/2.0 - -
 text/..
pywikipedia-newar4ur.py/r11135 Pywikipediabot/1.0 -
 -
Mozilla/5.0 (compatible; MyBot/1.0;) - -
 application/json
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
yasbot-gamar.py/r11054 Pywikipediabot/1.0 - -
 application/json
yasbot-gamar.py/r11054 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
nn-interwiki.py/r11060 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
nn-interwiki.py/r11027 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
XLinkBot/1.00 - -
 text/..
TjBot-1-interwiki.py/r11079 Pywikipediabot/1.0 - -
 application/json
 application/xml
nn-interwiki.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzredirectyeh.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
Friendly Spider 1.0 contact mail address - -
 text/..
pywikipedia-interwiki.py/r11060 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/x-www-form-urlencoded
pywikipedia2-interwiki.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11104 Pywikipediabot/1.0 - -
 application/json
 application/xml
 -
 application/x-www-form-urlencoded
1405693/r11081 Pywikipediabot/2.0 - -
 application/json
pywikipedia2-interwiki.py/r11085 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-zzredirectyeh.py/r11027 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pw2-interwiki.py/r11070 (wikipedia.py) Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
UCMore Crawler App en-us,en;q=0.5 -
 text/..
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot en-us,en;q=0.5 -
 text/..
nn-interwiki.py/r11110 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikt-interwiki.py/r11070 (wikipedia.py) Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
DotNetWikiBot/2.100 (Unix 3.2.0.35; ) - -
 text/..
1405693/r11081 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
UCI IR crawler 57484320 - -
 text/..
nn-interwiki.py/r11060 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia3-interwiki.py/r11071 Pywikipediabot/1.0 - -
 application/xml
 application/json
nn-interwiki.py/r11110 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-radeh.py/r11054 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-radeh.py/r11085 Pywikipediabot/1.0 -
 -
pywikt-interwiki.py/r11070 (wikipedia.py) Pywikipediabot/1.0 - -
 application/xml
 application/json
pywikipedia-radeh.py/r11085 Pywikipediabot/1.0 - -
 application/json
trunk-interwiki.py/r11114 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-radeh.py/r11054 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/r11055 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pw2-interwiki.py/r11070 (wikipedia.py) Pywikipediabot/1.0 - -
 application/xml
 application/json
theWxitBot/0.1 - -
 application/json
 image/..
 text/..
pynik-checkimages.py/r11054 Pywikipediabot/1.0 - -
 application/json
Semantix Bot 0.1 - -
 text/..
myurlcrawler/0.1 - -
 text/..
pywiki-main.py/r11071 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-info.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia2-interwiki.py/r11080 Pywikipediabot/1.0 - -
 application/json
 application/xml
ybot-ko_missing.py/r11099 Pywikipediabot/1.0 - -
 application/json
 application/xml
www.integromedb.org/Crawler
 text/..
 -
pynik-checkimages.py/r11085 Pywikipediabot/1.0 -
 -
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 416 subscribers; feed-id=3852576738117026533) - -
 application/xml
 -
pynik-checkimages.py/r11085 Pywikipediabot/1.0 - -
 application/json
SiteSeekerCrawler/1.0 - -
 text/..
 -
pywikipedia-info.py/r11027 Pywikipediabot/1.0 -
 -
rewrite-drp_warn.py/r11028 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
wikbotlite/2.0 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
 -
rewrite-drp_warn.py/r11028 Pywikipediabot/2.0 - -
 application/json
pynik-checkimages.py/r11054 Pywikipediabot/1.0 -
 -
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
trunk-interwiki.py/r11130 Pywikipediabot/1.0 - -
 application/xml
 application/json
COIBot/2.0 - -
 text/..
pywiki-main.py/r11071 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
trunk-interwiki.py/r11111 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
 application/ogg
pywikipedia-tasks.py/r10663 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/r11071 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 -
Empedia Bot - -
 text/..
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
pywikipedia-welcomeurdu.py/r11135 Pywikipediabot/1.0 -
 -
trunk-interwiki.py/r11085 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-commonscat.py/r11029 Pywikipediabot/1.0 - -
 application/json
pywikipedia-iwiki2_TS.py/r11131 Pywikipediabot/1.0 - -
 application/json
 application/xml
YourFilmsBot/0.1 - -
 application/json
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
nn-interwiki.py/r11102 Pywikipediabot/1.0 - -
 application/json
 application/xml
Geni ircpybot 1.0 -
 -
EarwigBot/0.2.dev.gitf082fca7 (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) -
 application/x-www-form-urlencoded
LaaknorBot-interwiki.py/r11071 Pywikipediabot/1.0 - -
 application/json
 application/xml
trunk-interwiki.py/r11055 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
Wikibot/2.0.2 CFNetwork/548.1.4 Darwin/11.0.0 en-us -
 image/..
 application/json
 text/..
HersfoldIRCBot version 1.5 - -
 -
 text/..
pywiki-commonscat.py/r11029 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-interwiki.py/r11058 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywiki-commonscat.py/r11097 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
ybot-ko_missing.py/r11114 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
EarwigBot/0.2.dev.gitf082fca7 (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
pynik-checkimages.py/r11060 Pywikipediabot/1.0 -
 -
pywikipedia-interwiki.py/r11085 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-interwiki.py/r11089 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-iwiki2_TS.py/r11131 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
python-wikitools/1.2 (User:Mr.Z-bot) - -
 application/json
pywikipedia-welcomeurdu.py/r11135 Pywikipediabot/1.0 - -
 application/json
pynik-checkimages.py/r11080 Pywikipediabot/1.0 -
 -
Find-My.com SearchSpider ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3 -
 text/..
trunk-interwiki.py/r11060 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
pywikipedia-wikt_wanted.py/r11027 Pywikipediabot/1.0 -
 -
pynik-checkimages.py/r11080 Pywikipediabot/1.0 - -
 application/json
nn-interwiki.py/r11102 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
SearchBot - -
 text/..
pywikipedia-wikt_wanted.py/r11027 Pywikipediabot/1.0 - -
 application/json
notify_disam-notify_disam/r11078 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (WikiProjectTagger; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia2-interwiki.py/r11102 Pywikipediabot/1.0 - -
 application/json
 application/xml
Wikibot/2.0.2 CFNetwork/609.1.4 Darwin/13.0.0 en-gb -
 image/..
 application/json
 text/..
DotNetWikiBot/2.101 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
pywikipedia-wikidata2.py/r11076 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11070 (wikipedia.py) Pywikipediabot/1.0 - -
 application/xml
 application/json
bot: fr-anal - -
 application/json
py2-myscan2.py/r11102 Pywikipediabot/1.0 - -
 application/json
python-wikitools/1.2 (User:Mr.Z-bot) -
 application/x-www-form-urlencoded
pywikipedia2-interwiki.py/r11131 Pywikipediabot/1.0 - -
 application/json
 application/xml
Geni ircpybot 1.0 - -
 application/json
 text/..
 application/xml
pywikipedia-interwiki.py/r11138 Pywikipediabot/1.0 - -
 application/json
 application/xml
Peachy MediaWiki Bot API Version 0.1beta -
 -
pynik-checkimages.py/r11060 Pywikipediabot/1.0 - -
 application/json
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.2/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
ybot-ko_missing.py/r11114 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia3-interwiki.py/r11029 Pywikipediabot/1.0 - -
 application/xml
 application/json
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
wAPI/1.1 (Bot: NoomBot Operator: Noommos Contact: mail address ) - -
 application/vnd.php.serialized
SuperBot/4.7.0.74 (Windows 7) - -
 text/..
python-wikitools/1.2 (User:LaraBot) -
 application/x-www-form-urlencoded
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
ZeBigWebBot - -
 text/..
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
pywiki-main.py/r11027 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-commonscat.py/r11029 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-commonscat.py/r11097 Pywikipediabot/1.0 - -
 application/json
 application/xml
python-wikitools/1.2 (User:BernsteinBot) - -
 application/json
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 -
 -
VWBot - CorenSearchBot/1.5 en derivative - -
 text/..
python-wikitools/1.2 (User:LaraBot) - -
 application/json
pywikipedia-hamsang.py/r11102 Pywikipediabot/1.0 - -
 application/json
pywikipedia3-interwiki.py/r11076 Pywikipediabot/1.0 - -
 application/xml
 application/json
pywikipedia2-interwiki.py/r11029 Pywikipediabot/1.0 - -
 application/json
 application/xml
DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
py2-myscan3.py/r11103 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0
 text/..
 -
pywikipedia-redirect.py/r11138 Pywikipediabot/1.0 - -
 application/json
opentask-opentasks.py/r10326 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
YBot/0.1 - -
 application/vnd.php.serialized
pywikipedia-basicreferenceslist.py/r11115 (wikipedia.py) Pywikipediabot/1.0 -
 -
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
LaaknorBot-interwiki.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11070 (wikipedia.py) Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-interwiki.py/r11110 Pywikipediabot/1.0 - -
 application/json
 application/xml
Peachy MediaWiki Bot API Version 0.1beta - -
 application/vnd.php.serialized
gsa-crawler (Enterprise; T3-F6UX6BBCD6SX6; mail address ) - -
 text/..
pywikipedia-28main.build84.py/r11131 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11081 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11088 Pywikipediabot/1.0 - -
 application/json
LaaknorBot-interwiki.py/r11085 Pywikipediabot/1.0 - -
 application/json
 application/xml
wikidata_blank_items/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
trunk-interwiki.py/r11102 Pywikipediabot/1.0 - -
 application/json
 application/xml
python-wikitools/1.2 (User:BernsteinBot) -
 application/x-www-form-urlencoded
OpenLink Virtuoso RDF crawler - -
 image/..
 text/..
trunk-interwiki.py/r11080 Pywikipediabot/1.0 - -
 application/json
 application/xml
LaaknorBot-interwiki.py/r11080 Pywikipediabot/1.0 - -
 application/json
 application/xml
MediaWiki::Bot 3.1.5 -
 application/x-www-form-urlencoded
pywikipedia-hamsang.py/r11102 Pywikipediabot/1.0 -
 -
Metabot 0.1 - -
 text/..
MediaWiki::Bot/v3.4.2 - -
 application/json
pywikipedia-redirect.py/r11113 Pywikipediabot/1.0 - -
 application/json
pywikipedia-commonscat.py/r11029 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-interwiki.py/r11029 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
GoogleBot
 text/..
 -
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
hercule-commonscat.py/r11080 Pywikipediabot/1.0 -
 -
pywikipedia-dikantenyvaovao.py/r10663 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
pywiki-main.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-main.py/r11029 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-bot_control.py/r494 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
Wikibot/2.0.2 CFNetwork/609.1.4 Darwin/13.0.0 de-de -
 image/..
 application/json
 text/..
user_scripts-llista1000.py/r11085 Pywikipediabot/1.0 -
 -
WorldOfMusicBetaBot/1.0 - -
 text/..
LauschenBot/1.1 ( mail address ) - -
 text/..
pywikipedia-interwiki.py/r11080 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
cd-delinker.py/r11029 Pywikipediabot/1.0 - -
 application/json
svnpywikipedia-interwiki.py/r11073 Pywikipediabot/1.0 - -
 application/json
 application/xml
Handelabra WikiBot - -
 text/..
 application/json
pywikipedia-welcome.py/r11071 Pywikipediabot/1.0 - -
 application/json
TjBot-1-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
pywikipedia-category_redirect.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
TjBot-1-interwiki.py/r11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-main.py/r11029 Pywikipediabot/1.0 - -
 application/json
 application/xml
Clueue Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
hercule-commonscat.py/r11130 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
35050.56total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Mar 11, 2013 15:49
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers