Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Dec 2012 - 14 Dec 2012 (last 12 months)
000 ⇒ k
 

 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 95,792,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 542,154,790 external requests, which is 17.7%

Page requests for crawlers that specify a url in the agent string
Count
x 1000
Secondary domain
(~site) name
URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cloudcrawling)
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
 code.google.com/p/crawler4j/text/..crawler4j (url)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen)
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url)
 code.google.com/appenginetext/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3)
 code.google.com/appenginetext/..Wiki.java 0.27 AppEngine-Google; (url; appid: wikipediatools)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~wikigraph2)
 code.google.com/appengineimage/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
 code.google.com/appengine-Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda)
 www.google.com/bot.htmlNONE/wikipedia- Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/oggMozilla/5.0 (compatible; GoogleBot/2.1; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox)
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki8)
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy0)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yourbudgets)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki1)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 114proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~theunblock)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy)
 code.google.com/appengineapplication/jsonPython-urllib/2.5 AppEngine-Google; (url; appid: loeschmonitor)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki6)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: your-zone)
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki7)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: usawebproxy0)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hr-pulsesubscriber)
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vi-mobile)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: goodersearch)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openeyeproxy)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki5)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline)
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khrixy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: guidesites)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp8)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh)
 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: vipoembed)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory)
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
 developers.facebook.comimage/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.phpapplication/jsonfacebookexternalhit/1.1 (url)
 developers.facebook.com-facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmapplication/vnd.php.serializedMozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-GoogleBot/2.1 (url)
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url)
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url)
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url)
 help.naver.com/robots/application/jsonYeti/1.0 (NHN Corp.; url)
msn
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url)
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/4.0; url)
 ahrefs.com/robot/application/oggMozilla/5.0 (compatible; AhrefsBot/4.0; url)
cibra
 cibra.de/text/..CiBra Data Collector (url)
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url)
finecomb
 finecomb.com/application/jsonapi/1.1 (url; mail address )
 finecomb.com/-api/1.1 (url; mail address )
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/-SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/application/oggMozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/screenshot/-Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url)
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url)
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 toolbar.youdao.com/image/..Youdao Toolbar (url)
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
 pear.php.net/text/..PEAR HTTP_Request class ( url )
 pear.php.net/image/..PEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8
 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
coccoc
 help.coccoc.vn/text/..coccoc/1.0 (url)
 help.coccoc.vn/-coccoc/1.0 (url)
wordpress
 fotosdeatrizesemodelos.wordpress.comtext/..WordPress/3.5-RC6-23166; url
 fotosdeatrizesemodelos.wordpress.comtext/..WordPress/3.5-RC5-23155; url
 lesliebrodie.wordpress.comtext/..WordPress/3.5-RC6-23166; url
 sreaves32.wordpress.comtext/..WordPress/3.5-RC4-23127; url
 josefboberg.wordpress.comtext/..WordPress/3.5-RC4-23127; url
 barbielistholland.wordpress.comtext/..WordPress/3.5-RC6-23166; url
 lesliebrodie.wordpress.comtext/..WordPress/3.5-RC4-23127; url
 josefboberg.wordpress.comtext/..WordPress/3.5-RC6-23166; url
 barbielistholland.wordpress.comtext/..WordPress/3.5-RC5-23155; url
 greatriversofhope.wordpress.comtext/..WordPress/3.5-RC6-23166; url
 unicorn144.wordpress.comtext/..WordPress/3.5-RC4-23127; url
 kbdto.wordpress.comtext/..WordPress/3.5-RC5-23141; url
 onanatomia.wordpress.comtext/..WordPress/3.5-RC5-23155; url
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
 www.image/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url)
 help.soso.com/webspider.htm-Mozilla/5.0(compatible; Sosospider/2.0; url)
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url)
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19 url
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.5.0-19-generic; java 1.7.0_09; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-45-server; java 1.6.0_26; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (Superarama-Beta/any; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.4.11-2.16-desktop; java 1.7.0_09; Europe/nl) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.6.4-1-ARCH; java 1.7.0_09; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_04; America/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.6.4-1-ARCH; java 1.7.0_09; Europe/fr) url
 yacy.net/bot.html-yacybot (freeworld/global; i386 Linux 3.5.0-19-generic; java 1.7.0_09; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.2; java 1.6.0_37; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-042stab061.2; java 1.7.0_09; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-1.16-default; java 1.6.0_24; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) url
 yacy.net/bot.html-yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.38-16-generic-pae; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86_64 Mac OS X 10.8.2; java 1.6.0_37; America/en) url
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 2.6.23.17-dbserv; java 1.6.0_04; Europe/de) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_37; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86 Windows 7 6.1; java 1.7.0_09; Europe/fr) url
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url)
yioop
 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url)
 www.yioop.com/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url)
wikidict
 www.wikidict.detext/..url
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url)
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url)
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url)
 toolserver.org/~dispenser/text/..DispensersTools (url)
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url)
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02
 toolserver.org/~dispenser/application/jsonDispensersTools (url)
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
dataparksearch
 dataparksearch.org/bottext/..DataparkSearch/4.54-26052011 (url)
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url)
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
okian
 www.okian.ro/text/..MyBot/1.0 (url)
wita
 www.wita.detext/..WITA/nutchbot/Nutch-1.5 (url; mail address )
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url)
 goo.gl/7y4SXtext/..GoogleProducer; (url)
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
 enwp.org/User:KingpinBottext/..KingpinBot (url)
 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
ephorus
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.4.5.6; url)
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url)
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
 SearchNearMe.com/contact.phptext/..SearchNearMe (url)
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url)
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
toshiba
 www.toshiba.co.jp/rdc/about/crawl_info.htmtext/..TosCrawler/Nutch-1.4 (url; ' mail address dot co dot jp')
 www.toshiba.co.jp/rdc/about/crawl_info.htmtext/..TosCrawler/Nutch-1.5.1 (url; ' mail address dot co dot jp')
cognarius
 cognarius.comapplication/jsonAppsArlak/1.0 (url)
 cognarius.comtext/..AppsArlak/1.0 (url)
discoveryengine
 discoveryengine.com/discoverybot.htmltext/..Mozilla/5.0 (compatible; discoverybot/2.0; url)
zeebox
 www.zeebox.comtext/..Zeebox (url)
 www.zeebox.comapplication/jsonZeebox (url)
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url)
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url)
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url)
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
 www.xbmc.orgtext/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
wikiglass
 wikiglass.comtext/..url : mail address
onet
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
easybib
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url)
 content.easybib.com/autocite/text/..EasyBib AutoCite (url)
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
 wikimpress.org/-Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
drupal
 drupal.org/image/..Drupal (url)
 drupal.org/text/..Drupal (url)
 drupal.org/text/..User-Agent: Drupal (url)
bibalex
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url)
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url)
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address )
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url)
 tineye.com/crawler.htmlimage/..TinEye/1.1 (url)
plos
 alm.plos.orgapplication/jsonPLoS Article Level Metrics - url
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address )
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
tiscali
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.10.2 url)
sf
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
friendofrenia
 friendofrenia.com/application/jsonUser-Agent: FriendoFrenia (url)
 friendofrenia.com/text/..User-Agent: FriendoFrenia (url)
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
 www.rcdtokyo.com/pc2m/-Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url)
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; Netseer crawler/2.0; url; mail address )
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url)
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url)
 www.clips.ua.ac.be/pages/patternapplication/jsonPattern/2.3 url
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url)
microsystools
 www.microsystools.com/products/sitemap-generator/text/..A1 Sitemap Generator/4.1.0 (url) miggibot
openindex
 www.openindex.io/en/webmasters/spider.htmltext/..Mozilla/5.0 (compatible; OpenindexSpider; url)
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url
 wiki.github.com/bixo/bixo/bixocrawlertext/..Mozilla/5.0 (compatible; pub-crawler; url; mail address )
kindsight
 www.kindsight.net/en/kscrawlertext/..KSCrawler/Nutch-1.0 (url; mail address )
 www.kindsight.net/en/kscrawlertext/..KSCrawler/Nutch-1.5.1 (url; mail address )
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Please get in touch if we are polling too hard.
 superfeedr.comtext/..Superfeedr bot/2.0 url - Please get in touch if we are polling too hard.
 superfeedr.com-Superfeedr bot/2.0 url - Please get in touch if we are polling too hard.
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url)
netvibes
 www.netvibes.comtext/..Netvibes (url)
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url)
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url)
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
avantbrowser
 www.avantbrowser.comtext/..Avant Browser (url)
 www.avantbrowser.comtext/..Advanced Browser (url)
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url)
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
thomasy
 map.thomasy.twapplication/jsonThomasy Map (url)
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
vermagerd
 www.vermagerd.be/wptext/..WordPress/3.4.2; url
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
elcidharth
 elcidharth.comtext/..WordPress/3.5-RC6-23166; url
 elcidharth.comtext/..WordPress/3.5-RC4-23127; url
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url)
newsgator
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
fotopedia
 www.fotopedia.comapplication/jsonPicor (url)
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
memidex
 www.memidex.com/_bottext/..Mozilla/5.0 (compatible; Memibot/1.0; url )
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
rockmelt
 rockmelt.comtext/..RockmeltEmbedService (url)
feedshow
 www.feedshow.comtext/..FeedshowOnline (url)
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2.1 (Feed Parser; url; Allow like Gecko) Build/20111015034325
 simplepie.orgtext/..SimplePie/1.2.1 (Feed Parser; url; Allow like Gecko) Build/20111015034325
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/3.1.0 url)
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url)
 duckduckgo.com/duckduckpreview.htmltext/..DuckDuckPreview/1.0; (url)
 duckduckgo.com/duckduckpreview.html-DuckDuckPreview/1.0; (url)
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
squirro
 intro.squirro.com/squirrobot/text/..squirrobot/1.0 (url)
 intro.squirro.com/squirrobot/image/..squirrobot/1.0 (url)
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url)
in
 www.m-culture.in.thtext/..m-culture.in.th (url)
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url)
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url)
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
Anonymouse
 Anonymouse.org/image/..url (Unix)
 Anonymouse.org/text/..url (Unix)
igrec
 www.igrec.ca/projectstext/..Wikitionary Text Parser 0.2 (url)
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
fueto
 fueto.comapplication/jsonFueto (url)
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url)
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url)
theworldtopbrands
 theworldtopbrands.comtext/..WordPress/3.4.2; url
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url)
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url)
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url)
wotbox
 www.wotbox.com/bot/text/..Wotbox/2.01 (url)
wiktionary
 en.wiktionary.org/wiki/User:Rukhabotapplication/jsonRukhabot/0.1 (url)
it-influentials
 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
zootycoon
 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
backgroundswitcher
 www.backgroundswitcher.com/text/..John's Background Switcher 4.6 (url)
 www.backgroundswitcher.com/image/..John's Background Switcher 4.4 (url)
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url)
mysite
 www.mysite.comtext/..Mozilla/5.0 (compatible; myAbstractCrawler url)
 www.mysite.comimage/..Mozilla/5.0 (compatible; myAbstractCrawler url)
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url
mysistermarilynmonroe
 mysistermarilynmonroe.orgtext/..WordPress/3.4.2; url
linguee
 www.linguee.com/bottext/..Linguee Bot (url; mail address )
holmes
 holmes.getext/..HolmesBot (url)
ranchero
 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
localhost
 localhosttext/..Mozilla/5.0 (compatible; heritrix/2.0.2 url)
blogbridge
 www.blogbridge.com/text/..BlogBridge 2.13 (url)
rssreader
 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
rssbandit
 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
nemui
 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
feeds4all
 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
127024.919999999total

Page requests for probable crawlers, recognized by keyword
Count
x 1000
Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0
 application/json
 application/xml
 text/..
 -
 application/x-www-form-urlencoded
 image/..
spider
 text/..
 application/vnd.php.serialized
 application/json
 -
 application/ogg
AniBot/0.9 php/curl
 application/vnd.php.serialized
 -
 text/..
php wikibot classes
 application/vnd.php.serialized
 text/..
 application/json
MediaWikiCrawler-Google/2.0 ( mail address )
 text/..
 -
GoogleBot-Image/1.0
 image/..
 text/..
 -
LinkParser/2.0
 text/..
 -
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
 text/..
SearchBot
 text/..
Peachy MediaWiki Bot API Version 1.0
 application/vnd.php.serialized
wikiwix-bot-3.0
 text/..
 -
GoogleBot-Image/1.0
 text/..
 image/..
 -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
 text/..
 -
 application/json
tigerbot
 application/json
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
Pywikipediabot/2.0
 application/json
ClueBot/1.1
 application/vnd.php.serialized
Answersbot
 text/..
ClueBot/2.0
 application/vnd.php.serialized
Wikipath Bot (email: mail address )
 application/json
TrueKnowledgeBot bot mail address >
 application/xml
 application/vnd.php.serialized
Mozilla 5.0 (Apibot 0.32)
 application/vnd.php.serialized
Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
 text/..
 application/json
 -
 image/..
DigitalsmithsBot
 text/..
User-Agent: (Researcher, Bot Newbie) .NET Bot, mail address
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
 image/..
 text/..
 -
 application/json
MediaWiki::Bot/3.2.6
 application/json
MediaWiki::Bot/3.005002
 application/json
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
 text/..
 -
 application/json
 image/..
plantspedia data crawler
 text/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
 application/json
www.integromedb.org/Crawler
 text/..
 image/..
 application/pdf
 application/ogg
Wikibot/2.0.1 CFNetwork/609 Darwin/13.0.0
 image/..
 application/json
 text/..
 -
mail address mail address – MediaWiki Tcl Bot Framework 0.5
 application/json
 application/x-www-form-urlencoded
mail address
 application/vnd.php.serialized
 text/..
YBot/0.1
 application/vnd.php.serialized
DotNetWikiBot/2.100 (Unix 2.6.32.38; )
 text/..
WikiPlaysBot
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 image/..
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
 text/..
 image/..
Tawbot (public svn release; plwiki)
 text/..
HN Spider/Nutch-2.1
 text/..
 application/ogg
Web Crawler
 text/..
www.monit24.pl-m24Bot/4.0-
 -
 image/..
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
 application/json
CorenSearchBot/1.7 en libwww-perl/6.04
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; )
 text/..
 application/xml
DotNetWikiBot/2.100 (Unix 5.10.0.0; )
 text/..
 application/xml
GermCrawler
 application/json
 text/..
dtSearchSpider
 text/..
SchoolReviewNetworkWikiBot
 application/json
SineBot/1.5.19(User:SineBot)
 application/vnd.php.serialized
 text/..
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/x-www-form-urlencoded
 application/xml
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
 application/json
My Nutch Spider/Nutch-1.5
 text/..
 image/..
 application/ogg
OrlodrimBot/1.0
 text/..
 -
 application/x-www-form-urlencoded
wikbotlite/2.0 CFNetwork/609 Darwin/13.0.0
 image/..
 application/json
 text/..
Phantom.js bot
 image/..
 text/..
HosiryuhosiBot IRC-RecentChanges Checker
 text/..
 application/x-www-form-urlencoded
Twitterbot/1.0
 text/..
 image/..
 -
 application/pdf
HTMLParser/1.6
 text/..
 -
Mozilla/5.0 (compatible; UnisterBot; mail address )
 text/..
 -
JavaCrawler/1.1
 text/..
MyBot ( mail address )
 text/..
MyCuteBot/0.1
 text/..
 application/json
Mozilla/5.0 (X11; Linux x86_64) Ubuntu/12.04 Codebot/1.0
 text/..
 image/..
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
 text/..
SiocWikiBot/1.0
 application/vnd.php.serialized
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
 application/json
 text/..
SurakWare MediaWiki Bot/1.0
 text/..
 application/xml
HTMLParser/2.0
 text/..
 -
HRoestBot, de-wikipedia using pywikipedia framework
 text/..
 application/json
DotNetWikiBot/2.101 (Unix 3.2.0.34; )
 text/..
Wikibot/2.0.1 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl
 text/..
 -
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
 application/json
 text/..
COIBot/1.00
 text/..
ToyStory Crawl uk.ac.dur.ddfw58 Dissertation Crawl
 text/..
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
 image/..
 text/..
Zing-BottaBot/2.0
 text/..
Test Webbot
 text/..
TVersity Media Robot
 text/..
parsijoo-crawler
 text/..
 application/ogg
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
 text/..
UCMore Crawler App
 text/..
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
 text/..
 -
XLinkBot/1.00
 text/..
GoogleBot
 text/..
 image/..
EarwigBot/0.2.dev.git4ff7612a (Python/2.7.3; https://github.com/earwig/earwigbot; mail address )
 application/json
 -
 text/..
 application/x-www-form-urlencoded
COIBot/2.0
 text/..
mySpider/Nutch-1.5.1
 text/..
IssueCrawler
 text/..
Metabot 0.1
 text/..
theWxitBot/0.1
 application/json
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
gsa-crawler (Enterprise; T3-F5C5JE7XKWWBK; mail address )
 text/..
Bot
 text/..
DotNetWikiBot/2.100 (Unix 3.0.0.12; )
 text/..
 application/xml
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
 text/..
Mozilla/5.0 (compatible; Tbot/1.0;)
 text/..
python-wikitools/1.2 (User:BernsteinBot)
 application/json
Anomebot v2.0
 application/json
 text/..
Peachy MediaWiki Bot API Version 0.1beta
 application/vnd.php.serialized
LauschenBot/1.0 ( mail address )
 text/..
DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
Nutch Spider/Nutch-1.5
 text/..
DotNetWikiBot/2.101 (Unix 3.1.9.0; )
 text/..
python-wikitools/1.2 (User:LaraBot)
 application/json
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
 application/xml
WikiBot/0.1
 text/..
 image/..
My Nutch Spider/Nutch-1.5.1
 text/..
 -
Mozilla/5.0 (Bgbot 0.5)
 text/..
MediaWiki::Bot/5.005004
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
 application/json
DotNetWikiBot/2.92 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
Rost WebSpider/1.00[cn](WinXP)
 text/..
 -
Empedia Bot
 text/..
SINA_ROBOT; Mozilla/5.0 (Windows; Windows NT 5.1; MSIE8.0; zh-CN; rv:1.9.1.8) Gecko/20100202 Firef8
 text/..
bitlybot
 text/..
 image/..
 -
Goalkeeperbot(User:Beetstra)/1.0
 text/..
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]])
 application/json
Mozilla 5.0 (Apibot 0.30b5)
 application/vnd.php.serialized
WPBot 1.0
 text/..
AdMedia bot
 text/..
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]])
 application/json
29595.81total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Sat, Mar 9, 2013 5:44
Author:Erik Zachte (
Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers