Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Jan 2013 - 30 Jan 2013 (last 12 months)
000 ⇒ k
 

 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 104,220,100 page requests (mime type text/html only!) per day are considered crawler requests, out of 594,396,100 external requests, which is 17.5%

Page requests for crawlers that specify a url in the agent string
Count
x 1000
Secondary domain
(~site) name
URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appengineapplication/jsonMozilla/5.0 AppEngine-Google; (url; appid: s~redconceptual) - -
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appenginetext/..Wiki.java 0.27 AppEngine-Google; (url; appid: wikipediatools) - -
 code.google.com/appenginetext/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appengine-Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img) - -
 www.google.com/bot.htmlNONE/wikipedia- Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: usawebproxy0) - -
 code.google.com/appenginetext/..Mozilla/5.0 AppEngine-Google; (url; appid: s~birthday-stats) - -
 code.google.com/appenginetext/..route-hacker alpha AppEngine-Google; (url; appid: s~routehackers) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app-cruxbot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 www.google.com/bot.htmlapplication/oggMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hr-pulsesubscriber) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~theunblock) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~liquid-helium) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
 code.google.com/appengineimage/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: your-zone) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) ja,en-us;q=0.7,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; script; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~italiatiki) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: azshaderserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azshaderserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-api) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
facebook
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url) - -
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en -
 www.bing.com/bingbot.htmapplication/vnd.php.serializedMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5 - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) en -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3 - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-GoogleBot/2.1 (url) en-us,en;q=0.5
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url)
 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url) ja, *;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
msn
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
 search.msn.com/msnbot.htmimage/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url)
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) tr,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
 cibra.de/text/..CiBra Data Collector (url)
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/reader/faq/topic006/-Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;url;1 subscribers;) - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/-Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
 fulltext.sblog.cz/screenshot/-Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
 fulltext.sblog.cz/-SeznamBot/3.0 (url)
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url) -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url)
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url)
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Sosospider(url)
 help.soso.com/webspider.htm-Sosospider(url)
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url)
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmimage/..Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Sosospider(url) - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17 - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
 pear.php.net/application/xmlPEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15 - -
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - -
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url)
 173.13.143.74/bot.php-Mozilla/5.0 (compatible; YioopBot; url) - -
 173.13.143.74/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url) - -
 173.13.143.74/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url)
 173.13.143.74/bot.php-Mozilla/5.0 (compatible; YioopBot; url)
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 wiki.toolserver.org/view/GeoHack-Geohack (url) -
 toolserver.org/~dispenser/-CacheThumbs/1.2 (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
 toolserver.org/~dispenser/-DispensersTools (url) -
 toolserver.org/~dispenser/application/x-www-form-urlencodedDispensersTools (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/application/jsonDispensersTools (url) - -
finecomb
 finecomb.com/-api/1.1 (url; mail address ) - -
 finecomb.com/application/jsonapi/1.1 (url; mail address ) - -
 finecomb.com/application/jsonapi/1.1 (url; mail address )
 finecomb.com/-api/1.1 (url; mail address )
wordpress
 josefboberg.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 tsjok45.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 kabarislam.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 jaggedykaye.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
 lesliebrodie.wordpress.comtext/..WordPress/3.6-alpha-23288; url - -
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.20.0 url - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url
 en.wikipedia.org/wiki/Wikipedia:Huggle-Huggle/2.1.19.0 url -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url)
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API) - -
 sk.wikipedia.org/wiki/Redaktor:TeslaBot-TeslaBot (url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url) zh-cn -
wikidict
 www.wikidict.detext/..url - -
 www.wikidict.detext/..url
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
 www.majestic12.co.uk/bot.php?-Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
coccoc
 help.coccoc.vn/text/..coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.vn/-coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.vn/text/..coccoc/1.0 (url)
 help.coccoc.vn/text/..coccoc/1.0 (url) - -
 help.coccoc.vn/text/..coccoc/1.0 (url) en-us,en;q=0.5 -
 help.coccoc.vn/-coccoc/1.0 (url)
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) - -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmlimage/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url) - -
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.35-minimal; java 1.7.0_10; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.6.11-1-ARCH; java 1.7.0_03; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 FreeBSD 9.1-RELEASE; java 1.7.0_011; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.6.11-1-ARCH; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-1.16-default; java 1.6.0_24; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_24; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.1; java 1.6.0_37; Asia/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-45-server; java 1.6.0_26; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.4-1-ARCH; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_07; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (torworld/any; amd64 Linux 3.4.0-5-generic; java 1.7.0_09; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; x86 Windows 7 6.1; java 1.6.0_18; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) url en-us,en;q=0.5 -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
cognarius
 cognarius.com-AppsArlak/1.0 (url) -
 cognarius.comapplication/jsonAppsArlak/1.0 (url) - -
 cognarius.comtext/..AppsArlak/1.0 (url) - -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.clips.ua.ac.be/pages/patternapplication/jsonPattern/2.3 url - -
 www.tkl.iis.u-tokyo.ac.jp/~crawler/-Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url) - -
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
 help.goo.ne.jp/contact/text/..goo wikipedia (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
 goo.gl/7y4SXimage/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
elcidharth
 elcidharth.comtext/..WordPress/3.6-alpha-23288; url - -
 elcidharth.comimage/..WordPress/3.6-alpha-23288; url - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
 labs.topsy.com/butterfly/-Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
 www.okian.ro/text/..MyBot/1.0 (url)
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
 www.gnip.com/text/..UnwindFetchor/1.0 (url)
 www.gnip.com/image/..UnwindFetchor/1.0 (url) - -
 www.gnip.com/-UnwindFetchor/1.0 (url) - -
gamedipper
 www.gamedipper.comapplication/jsongamedipper.com bot (url) - -
 www.gamedipper.comapplication/jsongamedipper.com bot (url)
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url) - -
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
 enwp.org/User:Hellknowz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET) - -
 enwp.org/User:KingpinBottext/..KingpinBot (url) - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url)
creativecloudlab
 creativecloudlab.com/CclWikipediaCrawler/text/..CclWikipediaCrawler/0.3 (url; mail address ) - -
 creativecloudlab.com/CclWikipediaCrawler/text/..CclWikipediaCrawler/0.1 (url; mail address ) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
 www.zeebox.comtext/..Zeebox (url)
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) af -
 bsurprised.com/-BSurprised WikiBox 0.1.3 (url) en
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) nl -
 bsurprised.com/application/x-www-form-urlencodedBSurprised WikiBox 0.1.3 (url) en
wikiglass
 wikiglass.comtext/..url : mail address - -
 wikiglass.comtext/..url : mail address
trendytrack
 trendytrack.comtext/..WordPress/3.5; url - -
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url) - -
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url) en-us,en;q=0.5 -
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url)
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
 www.speaktoit.comapplication/jsonSpeaktoit url
plos
 alm.plos.orgapplication/jsonPLoS Article Level Metrics - url - -
 alm.plos.org-PLoS Article Level Metrics - url -
 alm2-dev.plos.orgapplication/jsonArticle Level Metrics - url - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
 www.m-culture.in.thtext/..m-culture.in.th (url)
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) - -
 www.mediawiki.org/-MediaWiki OAI Harvester 0.2 (url) -
 www.mediawiki.org/wiki/Extension:RSStext/..MediaWikiRSS/0.02 (url) / MediaWiki RSS extension - -
fucinamediale
 labs.fucinamediale.comtext/..Mozilla/5.0 (compatible; ExperimentalWikiBot/1.0; url) - -
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url) - -
 www.xbmc.orgtext/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
 tineye.com/crawler.htmlimage/..TinEye/1.1 (url) - -
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url)
 tineye.com/crawler.htmltext/..TinEye/1.1 (url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20120911.190842 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) ja -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/-Drupal (url) -
 drupal.org/text/..Drupal (url)
 drupal.org/text/..User-Agent: Drupal (url) - -
bibalex
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url)
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url) - -
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url) - -
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
fivemost
 www.fivemost.comapplication/jsonurl - -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
 www.rcdtokyo.com/pc2m/-Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
 wikimpress.org/-Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
sf
 magpierss.sf.nettext/..MagpieRSS/0.7x (url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url) en-us,en;q=0.5 -
 liferea.sf.net/-Liferea/1.8.3 (Linux; fr_FR.UTF-8; url) -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
openindex
 www.openindex.io/en/webmasters/spider.htmltext/..Mozilla/5.0 (compatible; OpenindexSpider; url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.openindex.io/en/webmasters/spider.htmltext/..Mozilla/5.0 (compatible; OpenindexSpider; url) - -
localhost:8888
 localhost:8888image/..WordPress/3.5; url - -
 localhost:8888text/..WordPress/3.5; url - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
 www.moviecus.com/botcontactinfo.php-moviecus bot (url) -
example
 example.com/text/..Wiki 0.0 (url; mail address ) - -
 example.com/text/..WikiExample 0.1 (url; mail address ) - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
vermagerd
 www.vermagerd.be/wptext/..WordPress/3.4.2; url - -
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url) en-us,en;q=0.5 -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
scrapy
 scrapy.orgtext/..Scrapy/0.16.3 (url) en -
 scrapy.orgtext/..Scrapy/0.16.2 (url) en -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address ) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
publicknowledgeproject
 alm.publicknowledgeproject.orgapplication/jsonArticle Level Metrics - url - -
 alm.publicknowledgeproject.orgapplication/jsonArticle Level Metrics - url
rockmelt
 rockmelt.comtext/..RockmeltEmbedService (url) - -
n-grams
 www.n-grams.org/icorpusbot.htmltext/..iCorpusBot (url) es-es,en-us;q=0.7,en;q=0.3 -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url)
digplanet
 www.digplanet.com/application/vnd.php.serializedDigplanet/beta1 (url; mail address ) - -
avantbrowser
 www.avantbrowser.comtext/..Advanced Browser (url) en-us,en;q=0.5 -
 www.avantbrowser.comtext/..Avant Browser (url) en-us,en;q=0.5 -
simplepie
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
 superfeedr.com-Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
friendofrenia
 friendofrenia.com/text/..User-Agent: FriendoFrenia (url) - -
 friendofrenia.com/application/jsonUser-Agent: FriendoFrenia (url) - -
site-shot
 www.site-shot.com/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -
 www.site-shot.com/text/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url) en-us,en;q=0.5 -
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url) en-us,en;q=0.5 -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
onet
igrec
 www.igrec.ca/projectstext/..Wikitionary Text Parser 0.2 (url) - -
chickyrun
 chickyrun.tk/text/..ChickyBot/1.1 (url; mail address ) - -
newsgator
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP) en-us,en;q=0.5 -
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers) en-us,en;q=0.5 -
feedshow
 www.feedshow.comtext/..FeedshowOnline (url) en-us,en;q=0.5 -
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber) en-us,en;q=0.5 -
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -
metamagazine
 metamagazine.comtext/..WordPress/3.4.2; url - -
 metamagazine.com-WordPress/3.4.2; url -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 meta.stackoverflow.com/q/130398text/..Mozilla/5.0 (compatible; stackexchangebot/1.0; url) - -
dasdonkey
 www.dasdonkey.comtext/..Mozilla/5.0 (compatible; DonkeyBot/0.1; url) - -
linguee
 www.linguee.com/bottext/..Linguee Bot (url; mail address ) - -
 www.linguee.com/botapplication/jsonLinguee Bot (url; mail address ) - -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
medu
 medu.pl/bottext/..Mozilla/5.0 (compatible; Medubot/1.0; url) - -
 medu.pl/bot-Mozilla/5.0 (compatible; Medubot/1.0; url) -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
thomasy
 map.thomasy.twapplication/jsonThomasy Map (url) - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
localhost
 localhosttext/..Mozilla/5.0 (compatible; heritrix/2.0.2 url) - -
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.95; url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Bee-Python-urllib/2.7 (WikiApiary; Bumble Bee; url) -
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonPython-urllib/2.7 (WikiApiary; Bumble Bee; url) - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckpreview.htmltext/..DuckDuckPreview/1.0; (url) - -
 duckduckgo.com/duckduckpreview.html-DuckDuckPreview/1.0; (url) - -
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.4 (url) - -
 www.backgroundswitcher.com/image/..John's Background Switcher 4.6 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.3 (url) - -
wotbox
 www.wotbox.com/bot/text/..Wotbox/2.01 (url) - -
seomoz
 www.seomoz.org/dp/rogerbottext/..rogerbot/1.0 (url, mail address ) - -
 www.seomoz.org/dp/rogerbottext/..rogerbot/1.0 (url, mail address ) - -
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url) en-us -
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
blackfridayhighlights
 blackfridayhighlights.comtext/..WordPress/3.5; url - -
webceo
 online.webceo.comtext/..Mozilla/5.0 (compatible; web-ceo-online-bot/1.0; url) - -
theworldtopbrands
 theworldtopbrands.comtext/..WordPress/3.4.2; url - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
esciudad
 www.esciudad.com/application/jsonEsciudad/1.0 (url) - -
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9 en-us,en;q=0.5 -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -
weborking
 weborking.comtext/..Weborking(url) - -
 weborking.comapplication/jsonWeborking(url) - -
wikimedia
 tools.wikimedia.de/~para/GeoCommons/text/..url - -
 tools.wikimedia.de/~para/GeoCommons/-url -
blisshq
 www.blisshq.comimage/..bliss/20121128 url - -
 www.blisshq.comtext/..bliss/20121128 url - -
pinterest
 pinterest.com/image/..Pinterest/0.1 url - -
watchmouse
historicplaces
 www.historicplaces.net/text/..Monument Tool - Research USA historic Places - Saving wiki data in database for research - NON COMMERCIAL. (url - mail address ) - -
 www.historicplaces.net/application/jsonMonument Tool - Research USA historic Places - Saving wiki data in database for research - NON COMMERCIAL. (url - mail address ) - -
owasp
 www.owasp.org/index.php/Category:OWASP_DirBuster_Projecttext/..DirBuster-0.12 (url) - -
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
heartrails
 capture.heartrails.com/image/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.17 ja,en-us;q=0.7,en;q=0.3 -
 capture.heartrails.com/text/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.17 ja,en-us;q=0.7,en;q=0.3 -
compspy
 www.compspy.com/spider.htmltext/..Mozilla/5.0 (compatible; CompSpyBot/1.0; url) - -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.1 (url) - -
 www.yougorhymes.com/site/rhyme-bot-RhymeBot/0.1 (url) -
132771.689999989total

Page requests for probable crawlers, recognized by keyword
Count
x 1000
Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0 - -
 application/json
 application/xml
 text/..
 -
 application/x-www-form-urlencoded
 image/..
 application/pdf
PythonWikipediaBot/1.0 -
 -
 application/x-www-form-urlencoded
spider - -
 text/..
 application/vnd.php.serialized
 application/yaml
 application/json
 image/..
 -
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 -
 text/..
 image/..
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
LinkParser/2.0 - -
 text/..
DotNetWikiBot/2.101 (Unix 2.6.32.39; ) - -
 text/..
 application/xml
PythonWikipediaBot/1.0
 application/json
 application/xml
 -
 text/..
 application/x-www-form-urlencoded
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
 text/..
Peachy MediaWiki Bot API Version 1.0 -
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
wikiwix-bot-3.0 - -
 text/..
 -
tigerbot - -
 application/json
 text/..
 -
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
 text/..
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
 application/rsd+xml
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
 application/pdf
ClueBot/1.1 - -
 application/vnd.php.serialized
 text/..
Answersbot - -
 text/..
ClueBot/2.0 - -
 application/vnd.php.serialized
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider - -
 text/..
 application/xml
 application/json
 application/vnd.php.serialized
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
gsa-crawler (Enterprise; T3-P9JWVCTT9WWGY; mail address , mail address ) - -
 text/..
 -
spider
 text/..
 application/vnd.php.serialized
 application/json
www.integromedb.org/Crawler - -
 text/..
 -
 application/xml
 image/..
 application/ogg
Pywikipediabot/2.0 - -
 application/json
 text/..
TrueKnowledgeBot bot mail address > - -
 application/xml
 application/vnd.php.serialized
 image/..
Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
Wikipath Bot (email: mail address ) - -
 application/json
Mozilla 5.0 (Apibot 0.32) - -
 application/vnd.php.serialized
 text/..
php wikibot classes
 application/vnd.php.serialized
 text/..
gsa-crawler (Enterprise; T3-P9JWVCTT9WWGY; mail address ) - -
 text/..
DigitalsmithsBot - -
 text/..
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 -
 image/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
MediaWikiCrawler-Google/2.0 ( mail address )
 text/..
 -
crawl/0.4 libcrawl/0.3 - -
 text/..
 image/..
 application/json
 application/ogg
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
 image/..
 application/ogg
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
XBot v1.0 using MER-C's Wiki.java - -
 text/..
gsa-crawler (Enterprise; T3-P9JWVCTT9WWGY; mail address , mail address )
 text/..
 -
GoogleBot-Image/1.0
 image/..
 text/..
 -
MediaWiki::Bot/3.2.6 - -
 application/json
 -
 text/..
LinkParser/2.0
 text/..
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
SearchBot - -
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
WikiCatResearchBot ( mail address ) - -
 text/..
Tawbot (public svn release; plwiki) - -
 text/..
mail address - -
 application/vnd.php.serialized
 text/..
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
 -
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
 application/x-www-form-urlencoded
AarghBot Linux - -
 text/..
HTMLParser/1.6 - -
 text/..
 -
gsa-crawler (Enterprise; T3-F5C5JE7XKWWBK; mail address ) - -
 text/..
 -
DotNetWikiBot/2.101 (Unix 2.6.32.39; )
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
 image/..
 application/ogg
www.monit24.pl-m24Bot/4.0- - -
 -
 image/..
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 -
SchoolReviewNetworkWikiBot - -
 application/json
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
ExactusBot-v0.1 - -
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.35; ) - -
 text/..
Hulubot - -
 text/..
wikiwix-bot-3.0
 text/..
 -
AniBot/0.9 php/curl
 application/vnd.php.serialized
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
GoogleBot-Image/1.0
 text/..
 image/..
 -
dtSearchSpider - -
 text/..
Peachy MediaWiki Bot API Version 1.0
 application/vnd.php.serialized
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
 text/..
GermCrawler - -
 application/json
 text/..
php wikibot classes -
 -
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; ) - -
 text/..
 application/xml
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
 text/..
SearchBot
 text/..
 -
www.integromedb.org/Crawler
 text/..
Test Webbot - -
 text/..
 application/json
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
YBot/0.1 - -
 application/vnd.php.serialized
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
mySpider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
Phantom.js bot cs-CZ,en,* -
 image/..
 text/..
 -
MediaWiki::Bot/3.005002 - -
 application/json
- Peachy MediaWiki Bot API Version 1.0 -
 multipart/form-data
tigerbot
 application/json
 text/..
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
360spider-image - -
 image/..
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
 text/..
 -
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
 application/x-www-form-urlencoded
Twitterbot/1.0 - -
 text/..
 image/..
 -
 application/pdf
JavaCrawler/1.1 - -
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
SurakWare MediaWiki Bot/1.0 - -
 text/..
 application/xml
Answersbot
 text/..
milog_bot/1.0 ( mail address ) - -
 text/..
- PythonWikipediaBot/1.0 -
 multipart/form-data
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
Pywikipediabot/2.0
 application/json
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
DotNetWikiBot/2.100 (Unix 2.6.32.38; ) - -
 text/..
Spinuf Spider - -
 text/..
 -
ClueBot/1.1
 application/vnd.php.serialized
Wikipath Bot (email: mail address )
 application/json
GoogleBot-Image/1.0 -
 -
 image/..
wikiwix-bot-3.0 en -
 text/..
ClueBot/2.0
 application/vnd.php.serialized
OrlodrimBot/1.0 - -
 text/..
 -
 application/x-www-form-urlencoded
HRoestBot, de-wikipedia using pywikipedia framework -
 application/x-www-form-urlencoded
 -
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) - -
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 image/..
 text/..
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
crawler - -
 image/..
 text/..
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) -
 -
Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl - -
 text/..
 -
XLinkBot/1.00 - -
 text/..
HTMLParser/2.0 - -
 text/..
 -
 application/xml
TicketsBot/0.1 - -
 text/..
WPBot 1.0 - -
 text/..
 image/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
Zing-BottaBot/2.0 - -
 text/..
WikiBot/0.1 - -
 text/..
 image/..
TVersity Media Robot - -
 text/..
DigitalsmithsBot
 text/..
Mozilla 5.0 (Apibot 0.32)
 application/vnd.php.serialized
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
 image/..
Junut Bot 1.0.3 ru-RU,en,* -
 text/..
 -
 application/xml
wikbotlite/2.0 CFNetwork/609 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
YBot/0.1
 application/vnd.php.serialized
Empedia Bot - -
 text/..
Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
 text/..
 application/json
 image/..
COIBot/1.00 - -
 text/..
UCMore Crawler App en-us,en;q=0.5 -
 text/..
theWxitBot/0.1 - -
 application/json
 image/..
FAST Search Web Crawler 14.0.0325.0000 -
 -
 text/..
FAST Search Web Crawler 14.0.0325.0000 - -
 text/..
 -
 image/..
 application/ogg
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0
 image/..
 application/json
 text/..
DotNetWikiBot/2.101 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot en-us,en;q=0.5 -
 text/..
DotNetWikiBot/2.101 (Unix 2.6.32.45; ) - -
 text/..
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; )
 text/..
Metabot 0.1 - -
 text/..
MediaWiki::Bot/3.2.6
 application/json
MBBot/1.0.0 en-us,en;q=0.5 -
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
 image/..
 text/..
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1) - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 416 subscribers; feed-id=3852576738117026533) - -
 application/xml
 -
python-wikitools/1.2 (User:BernsteinBot) - -
 application/json
Geni ircpybot 1.0 - -
 text/..
 application/json
 application/xml
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
Wikibot/2.0.2 CFNetwork/548.1.4 Darwin/11.0.0 en-us -
 image/..
 application/json
 text/..
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
mail address mail address – MediaWiki Tcl Bot Framework 0.5
 application/json
 application/x-www-form-urlencoded
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 -
COIBot/2.0 - -
 text/..
python-wikitools/1.2 (User:BernsteinBot) -
 application/x-www-form-urlencoded
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1) -
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
mail address
 application/vnd.php.serialized
 text/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
 application/json
TrueKnowledgeBot bot mail address >
 application/xml
 application/vnd.php.serialized
HersfoldIRCBot version 1.2.1 - -
 text/..
 -
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
Geni ircpybot 1.0 -
 -
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
Mozilla/5.0 (compatible; NewThingForMeBot; mailto: mail address ) - -
 text/..
 image/..
Wikibot/2.0.1 CFNetwork/609 Darwin/13.0.0 en-us -
 image/..
 application/json
 text/..
 -
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 image/..
python-wikitools/1.2 (User:LaraBot) - -
 application/json
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-gb -
 image/..
 application/json
 text/..
 -
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 de-de -
 image/..
 application/json
 text/..
 -
python-wikitools/1.2 (User:LaraBot) -
 application/x-www-form-urlencoded
IsraBot - -
 text/..
lexiapp-crawler/0.1.0 ( mail address )
 text/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.2/1.8.5.168;) Opera Mini/3.1 - -
 image/..
 text/..
 -
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
MerlBot -
 -
GoogleBot - -
 text/..
 -
OpenLink Virtuoso RDF crawler - -
 image/..
 text/..
 -
LWNutch/Nutch-1.4 (another scientific bot - we accept your robots.txt! ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Mozilla/5.0 (bot; user:amalthea; en-us) - -
 text/..
 -
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
LauschenBot/1.1 ( mail address ) - -
 text/..
Peachy MediaWiki Bot API Version 0.1beta -
 -
Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 zh-cn -
 image/..
 application/json
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 2; ) - -
 text/..
bitlybot - -
 text/..
 image/..
 -
LauschenBot/1.1 ( mail address ) -
 -
FAST Enterprise Crawler 6 used by contosoa.com ( mail address ) - -
 text/..
MerlBot - -
 application/vnd.php.serialized
bot: fr-anal - -
 application/json
VWBot - CorenSearchBot/1.5 en derivative - -
 text/..
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
Peachy MediaWiki Bot API Version 0.1beta - -
 application/vnd.php.serialized
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
MediaWiki::Bot 3.1.5 -
 application/x-www-form-urlencoded
37851.06total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Sat, Mar 9, 2013 16:23
Author:Erik Zachte (
Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers