Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Sep 2012 - 30 Sep 2012 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 81,462,200 page requests (mime type text/html only!) per day are considered crawler requests, out of 495,754,330 external requests, which is 16.4%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cloudcrawling)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~syytacit)
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hr-pulsesubscriber)
 code.google.com/p/crawler4j/text/..crawler4j (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~edurep-metadata-quality)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
 code.google.com/appengineimage/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url)
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~wikigraph2)
 code.google.com/p/rondaapplication/jsonRonda - url
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
 code.google.com/p/rondatext/..Ronda - url
 code.google.com/appenginetext/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kurizogeorge)
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 code.google.com/appengineapplication/jsonMozilla 4.0 AppEngine-Google; (url; appid: prfleme)
 code.google.com/appengineapplication/jsonMozilla 3.5 AppEngine-Google; (url; appid: prfleme)
 code.google.com/appenginetext/..Wiki.java 0.26 AppEngine-Google; (url; appid: wikipediatools)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: usawebproxy0)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: prfleme)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy0)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda)
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-eye-ear)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10)
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 developers.facebook.comimage/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.phpapplication/jsonfacebookexternalhit/1.1 (url)
 developers.facebook.com-facebookplatform/1.0 (url)
 developers.facebook.comtext/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url)
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Nano; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url)
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexNews/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url)
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
 corp.naver.jp/text/..Mozilla/5.0 (compatible; NaverJapan/1.0; url)
 help.naver.com/robots/application/jsonYeti/1.0 (NHN Corp.; url)
msn
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url)
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htm-msnbot/2.0b (url)
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url)
cibra
 cibra.de/text/..CiBra Data Collector (url)
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 toolbar.youdao.com/image/..Youdao Toolbar (url)
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/-SeznamBot/3.0 (url)
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
toolserver
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url)
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url)
 toolserver.org/~dispenser/text/..DispensersTools (url)
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02
 toolserver.org/~dispenser/application/jsonDispensersTools (url)
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
 pear.php.net/text/..PEAR HTTP_Request class ( url )
 pear.php.net/image/..PEAR HTTP_Request class ( url )
 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible; Sosospider/2.0; url)
 help.soso.com/webspider.htm-Mozilla/5.0(compatible; Sosospider/2.0; url)
 help.soso.com/webspider.htmapplication/jsonMozilla/5.0(compatible; Sosospider/2.0; url)
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
 www.image/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
wordpress
 josefboberg.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 support.wordpress.com/contact/text/..WordPress.com mShots; url
 imagenssagradas.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 02varvara.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 wildanrenaldi.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 greatriversofhope.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 driwancybermuseum.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 iwansuwandy.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 klausgauger.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 tsjok45.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 midnightduke8.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 gunnyg.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 jamesmessig.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 playallgalaxies.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 factoriahistorica.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 investigationsoanisetoceanographiee.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 einflussreicheleute.wordpress.comtext/..WordPress/3.5-alpha-21535; url
 villatuelda.wordpress.comtext/..WordPress/3.5-alpha-21535; url
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url)
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
discoveryengine
 discoveryengine.com/discoverybot.htmltext/..Mozilla/5.0 (compatible; discoverybot/2.0; url)
 discoveryengine.com/discoverybot.html-Mozilla/5.0 (compatible; discoverybot/2.0; url)
 discoveryengine.com/discoverybot.htmlimage/..Mozilla/5.0 (compatible; discoverybot/2.0; url)
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url)
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url)
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_02; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_02; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_07; Europe/es) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-29-generic; java 1.7.0_03; Europe/sv) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-30-generic; java 1.7.0_07; GMT01:00/sv) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_07; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-42-generic; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-30-generic; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_05; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.2; java 1.6.0_35; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-29-generic; java 1.6.0_24; Indian/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-28-generic; java 1.6.0_20; Europe/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-29-generic; java 1.6.0_24; Indian/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-32-generic; java 1.6.0_20; Europe/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_07; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.23.17-dbserv; java 1.6.0_04; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-32-generic; java 1.6.0_20; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-43-generic; java 1.6.0_24; Europe/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-30-generic; java 1.7.0_07; GMT01:00/sv) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_33; UTC/en) url
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url)
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url)
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url)
cognarius
 cognarius.comapplication/jsonAppsArlak/1.0 (url)
 cognarius.comtext/..AppsArlak/1.0 (url)
localhost
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
medwhat
 www.medwhat.com/application/jsonMedWhatCrawler/1.1 (url; mail address ) Java/1.7.0_04
 www.medwhat.com/image/..MedWhatCrawler/1.1 (url; mail address ) Java/1.7.0_04
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url)
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
wikidict
 www.wikidict.detext/..url
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
 mymemory.traslated.net/doc/-Mozilla/5.0 (MyMemory Bot url)
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url)
 goo.gl/7y4SXtext/..GoogleProducer; (url)
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
 goo.gl/7y4SXimage/..GoogleProducer; (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
microsystools
 www.microsystools.com/products/sitemap-generator/text/..A1 Sitemap Generator/3.5.1 (url) miggibot
 www.microsystools.com/products/sitemap-generator/image/..A1 Sitemap Generator/3.5.1 (url) miggibot
 www.microsystools.com/products/website-download/text/..A1 Website Download/2.1.3 (url) miggibot
 www.microsystools.com/products/website-download/image/..A1 Website Download/2.1.3 (url) miggibot
coccoc
 help.coccoc.vn/text/..coccoc/1.0 (url)
 help.coccoc.vn/-coccoc/1.0 (url)
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
 enwp.org/User:KingpinBottext/..KingpinBot (url)
 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
dasdonkey
 www.dasdonkey.comtext/..Mozilla/5.0 (compatible; DonkeyBot/0.1; url)
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url)
whatrhymeswith
 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
okian
 www.okian.ro/text/..MyBot/1.0 (url)
ephorus
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.4.5.6; url)
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.4.5.5; url)
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.4.5.4; url)
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.4.5.1; url)
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url)
 www.gnip.com/-UnwindFetchor/1.0 (url)
 www.gnip.com/image/..UnwindFetchor/1.0 (url)
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url)
plos
 alm.plos.orgapplication/jsonPLoS Article Level Metrics - url
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
 SearchNearMe.com/contact.phptext/..SearchNearMe (url)
nettenis
 www.nettenis.orgtext/..WordPress/3.4.2; url
 www.nettenis.orgimage/..WordPress/3.4.2; url
wikiglass
 wikiglass.comtext/..url : mail address
freebase
 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
 textdigger.comimage/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
svglib
 svglib.orgtext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 svglib.org-Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 svglib.orgimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url)
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
fucinamediale
 labs.fucinamediale.comtext/..Mozilla/5.0 (compatible; ExperimentalWikiBot/1.0; url)
sf
 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address )
parsijoo
 www.parsijoo.irtext/..Mozilla/5.0 (compatible; mail address url)
 parsijoo.irtext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
numberfound
 www.numberfound.it/text/..MyBot/1.0 (url)
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url)
 www.proximic.comtext/..Mozilla/5.0 (compatible; proximic; url)
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url)
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
sciencecard
 demo.sciencecard.orgapplication/jsonArticle Level Metrics - url
drupal
 drupal.org/image/..Drupal (url)
 drupal.org/text/..Drupal (url)
 drupal.org/text/..User-Agent: Drupal (url)
 drupal.org/application/xmlDrupal (url)
 drupal.org/-Drupal (url)
 drupal.org/text/..Drupal (url
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
 wikimpress.org/-Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
wiktionary
 en.wiktionary.org/wiki/User:Rukhabotapplication/jsonRukhabot/0.1 (url)
 en.wiktionary.org/wiki/User:Daneelapplication/jsonDaneel Olivaw/0.1 (url Olivaw)
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address )
tineye
 tineye.com/crawler.htmlimage/..TinEye/1.1 (url)
 tineye.com/crawler.htmltext/..TinEye/1.1 (url)
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url)
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url)
toshiba
 www.toshiba.co.jp/rdc/about/crawl_info.htmtext/..TosCrawler/Nutch-1.4 (url; ' mail address dot co dot jp')
 www.toshiba.co.jp/rdc/about/crawl_info.htmtext/..TosCrawler/Nutch-1.5.1 (url; ' mail address dot co dot jp')
tiscali
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.01.18 url)
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
 github.com/edsu/wikitweetsapplication/jsonwikitweets <url
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.95; url)
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
avantbrowser
 www.avantbrowser.comtext/..Avant Browser (url)
 www.avantbrowser.comtext/..Advanced Browser (url)
feedshow
 www.feedshow.comtext/..FeedshowOnline (url)
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
newsgator
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
friendofrenia
 friendofrenia.com/application/jsonUser-Agent: FriendoFrenia (url)
 friendofrenia.com/text/..User-Agent: FriendoFrenia (url)
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
worldaswillandfarce
 worldaswillandfarce.comtext/..WordPress/3.5-alpha-21535; url
249
 173.212.249.18/bot.phptext/..Mozilla/5.0 (compatible; Woozie! Crawler; url)
 173.212.249.18/bot.phpimage/..Mozilla/5.0 (compatible; Woozie! Crawler; url)
 173.212.249.18/yioop-v0.90/bot.phptext/..Mozilla/5.0 (compatible; Woozie! Crawler; url)
tumblr
 benderthewebrobot.tumblr.comtext/..Mozilla/5.0 (compatible; Bender; url)
kalooga
 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url)
 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url)
enotes
 www.enotes.comtext/..eNotesBot 2.0 (url)
 www.enotes.comimage/..eNotesBot 2.0 (url)
thearchangelmichael
 thearchangelmichael.nettext/..WordPress/3.4.1; url
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2.1 (Feed Parser; url; Allow like Gecko) Build/20111015034325
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
 simplepie.orgtext/..SimplePie/1.2.1 (Feed Parser; url; Allow like Gecko) Build/20111015034325
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/3.3.2 (Mac OS X; url; gzip-happy)
pagefreezer
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address )
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address )
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
quaba
 quaba.detext/..quaba spider (url deutsche Suchmaschine)
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url)
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url)
weblio
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
grid-son
 grid-son.comapplication/jsonurl
silverfiresoftware
 www.silverfiresoftware.com/pyrokeettext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.silverfiresoftware.comimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.silverfiresoftware.com-Mozilla/5.0 (compatible; heritrix/1.14.4 url)
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url
cheapdealsx
example
 example.com/MyCoolToolPage/application/vnd.php.serializedUser-Agent: MyCoolTool (url)
pagepeeker
 pagepeeker.com/robots/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 KHTML Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; url
 pagepeeker.com/robots/text/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 KHTML Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; url
matuschek
 www.matuschek.net/jobo.htmltext/..JoBo/1.4 (url)
search
 www.search.ch/rim.htmltext/..UltraSpider3000/1.0 (url)
mobileproxy
 mobileproxy.mobitext/..Mozilla/5.0 (compatible; MobileSurf; url)
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
blogbridge
 www.blogbridge.com/text/..BlogBridge 2.13 (url)
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
 www.sonyericsson.com/UAprof/R800xR301.xmltext/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url)
 duckduckgo.com/duckduckpreview.htmltext/..DuckDuckPreview/1.0; (url)
 duckduckgo.com/duckduckpreview.html-DuckDuckPreview/1.0; (url)
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
netvibes
 www.netvibes.comtext/..Netvibes (url)
feeds4all
 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
kula
 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
orcabrowser
 www.orcabrowser.comtext/..Orca Browser (url)
snarfware
 www.snarfware.com/text/..Snarfer/0.x.x (url)
zootycoon
 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
aodblog
 aodblog.comtext/..WordPress/3.3.1; url
rssreader
 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
timewe
 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url)
ranchero
 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
fotopedia
 www.fotopedia.comapplication/jsonPicor (url)
rssbandit
 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
quus
 quus.net/text/..url
it-influentials
 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
winpodder
 winpodder.comtext/..WinPodder (url)
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
zipcommander
 www.zipcommander.com/text/..1st ZipCommander (Net) - url
ponderer
 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
seebot
 seebot.orgtext/..Lynx/2.8 (;url)
dealsbestz
graemef
 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
nemui
 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
plagger
 plagger.org/text/..Plagger/0.x.xx (url)
everythingwiki
 everythingwiki.net/EverythingWikiBot.htmltext/..Mozilla/5.0 (compatible; EverythingWikiBot/0.2; url)
 everythingwiki.net/EverythingWikiBot.htmltext/..Mozilla/5.0 (compatible; EverythingWikiBot/0.1; url)
sencha
 www.sencha.com/products/io/text/..Sencha.io-Src; (url)
 www.sencha.com/products/io/image/..Sencha.io-Src; (url)
sourceforge
 fess.sourceforge.jp/bot.htmltext/..Mozilla/5.0 (compatible; Fess/7.0; url)
 linkchecker.sourceforge.net/text/..LinkChecker/7.9 (url)
networkedblogs
 www.networkedblogs.comimage/..NetworkedBlogs (url;) AppEngine-Google; (http://code.google.com/appengine; appid: s~networkedblogshr)
metamagazine
 metamagazine.comtext/..WordPress/3.4.2; url
mysistereileen
 mysistereileen.comtext/..WordPress/3.4.2; url
 mysistereileen.comtext/..WordPress/3.4.1; url
 mysistereileen.orgtext/..WordPress/3.4.2; url
Anonymouse
 Anonymouse.org/image/..url (Unix)
 Anonymouse.org/text/..url (Unix)
superfeedr
 superfeedr.comapplication/xmlSuperfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
globalspec
 www.globalspec.com/Ocellitext/..Ocelli/1.4 (url)
126196.499999991total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0
 application/json
 application/xml
 -
 text/..
 application/x-www-form-urlencoded
 image/..
spider
 text/..
 application/vnd.php.serialized
 -
MediaWikiCrawler-Google/2.0 ( mail address )
 text/..
 -
GoogleBot-Image/1.0
 image/..
 text/..
 -
php wikibot classes
 application/vnd.php.serialized
 -
 text/..
LinkParser/2.0
 text/..
 -
AniBot/0.9 php/curl
 application/vnd.php.serialized
 -
 image/..
GoogleBot-Image/1.0
 text/..
 image/..
 -
 application/json
Peachy MediaWiki Bot API Version 1.0
 application/vnd.php.serialized
wikiwix-bot-3.0
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
 text/..
 application/json
 -
Pywikipediabot/2.0
 application/json
 text/..
ClueBot/1.1
 application/vnd.php.serialized
Answersbot
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
 text/..
 -
 image/..
 application/json
 application/xml
 application/pdf
 application/ogg
ClueBot/2.0
 application/vnd.php.serialized
Mozilla 5.0 (Apibot 0.32)
 application/vnd.php.serialized
Wikipath Bot (email: mail address )
 application/json
plantspedia data crawler
 text/..
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
cleaner-wikipedia bot / self.maluke.com
 application/json
 text/..
tigerbot
 application/json
 text/..
DigitalsmithsBot
 text/..
mail address
 application/vnd.php.serialized
 text/..
NexiSpider/Nutch-1.5.1
 text/..
 -
MediaWiki::Bot/3.2.6
 application/json
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
 text/..
 application/json
 application/xml
 image/..
 application/vnd.php.serialized
 -
wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 image/..
 audio/midi
python-wikitools/1.2 (User:BernsteinBot)
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
 image/..
 text/..
 -
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; )
 text/..
 application/xml
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
Tawbot (public svn release; plwiki)
 text/..
Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl
 text/..
 -
DotNetWikiBot/2.100 (Unix 2.6.32.38; )
 text/..
 application/xml
 -
Bot building hyperlink map --- mail address
 text/..
Webwiki Search Engine Bot - www.webwiki.de
 text/..
SineBot/1.5.19(User:SineBot)
 application/vnd.php.serialized
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1)
 application/json
 text/..
MLBot (www.metadatalabs.com/mlbot)
 text/..
 application/vnd.php.serialized
 application/json
 -
JavaCrawler/1.1
 text/..
FAST Search Web Crawler 14.0.0325.0000
 text/..
 -
 application/xml
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
GermCrawler
 application/json
 text/..
www.integromedb.org/Crawler
 text/..
 application/xml
MediaWiki::Bot/3.005002
 application/json
wikbot/1.60 CFNetwork/609 Darwin/13.0.0
 image/..
 application/json
 text/..
 -
Test Webbot
 text/..
 -
DotNetWikiBot/2.100 (Unix 5.10.0.0; )
 text/..
 application/xml
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
 application/json
Perl's Analytic Bot/1.0
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
 text/..
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
 application/json
SchoolReviewNetworkWikiBot
 application/json
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
 application/json
 text/..
HTMLParser/2.0
 text/..
 -
 image/..
User-Agent: eGexaBot
 text/..
 application/json
SearchBot
 text/..
Twitterbot/1.0
 text/..
 image/..
 -
 application/pdf
 application/ogg
SiocWikiBot/1.0
 application/vnd.php.serialized
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
ZutopiBot/Nutch-1.5.1
 text/..
HosiryuhosiBot IRC-RecentChanges Checker
 text/..
 application/x-www-form-urlencoded
PracticeBot/0.2 (Testing)
 text/..
 -
wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
OrlodrimBot/1.0
 text/..
 -
 application/x-www-form-urlencoded
Mozilla/5.0 crawler/suggest.io
 text/..
 -
SurakWare MediaWiki Bot/1.0
 text/..
 application/xml
mySpider/Nutch-1.5.1
 text/..
 -
AdMedia bot
 text/..
COIBot/1.00
 text/..
DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
UCMore Crawler App
 text/..
 -
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
 text/..
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/x-www-form-urlencoded
 application/xml
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
 image/..
 text/..
 -
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
 text/..
TVersity Media Robot
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5
 application/json
 application/x-www-form-urlencoded
VWBot - CorenSearchBot/1.5 en derivative
 text/..
YBot/0.1
 application/vnd.php.serialized
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
 application/json
 text/..
HRoestBot, de-wikipedia using pywikipedia framework
 text/..
 application/json
TwynCatBot/0.1 (Contact: www.twyn.com)
 application/json
bitlybot
 text/..
 image/..
 -
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
 application/xml
 text/..
infraEnterprise v8 Web Crawler
 -
 text/..
GoogleBot
 text/..
 image/..
Zing-BottaBot/2.0
 text/..
CorenSearchBot/1.7 en libwww-perl/6.02
 text/..
FTRF: Friendly robot/1.3
 text/..
 -
 application/xml
Knownet Bot/0.1.0
 text/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
YoonoCrawler/1.0 ( mail address )
 text/..
Doddebot
 text/..
DotNetWikiBot/2.100 (Unix 3.0.0.12; )
 text/..
 application/xml
LauschenBot/1.0 ( mail address )
 text/..
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
 text/..
 -
MyCuteBot/0.1
 text/..
 application/json
 application/vnd.php.serialized
Phantom.js bot
 image/..
 text/..
COIBot/2.0
 text/..
Mozilla/5.0 (Bgbot 0.5)
 text/..
Surag Spider/Nutch-1.4
 text/..
 image/..
Bot
 text/..
myrobot
 text/..
Mozilla/5.0 (compatible; Tbot/1.0;)
 text/..
Analytic Bot/1.0
 application/json
Crawler/Nutch-1.4
 text/..
 -
Mozilla/5.0 (compatible; UnisterBot; mail address )
 text/..
 application/ogg
Goalkeeperbot(User:Beetstra)/1.0
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7600.0; )
 text/..
Wiki.java 0.26 (OctraBot 1.5)
 text/..
 -
 application/x-www-form-urlencoded
KumulBot/0.24
 application/vnd.php.serialized
WPBot 1.0
 text/..
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
 application/xml
ClueBot/2.0 (ClueBot NG Report Interface)
 text/..
XLinkBot/1.00
 text/..
MediaWiki::Bot/1.00
 text/..
 -
TrueKnowledgeBot bot mail address >
 application/xml
 application/vnd.php.serialized
 text/..
Wikibot 1.54 (Macintosh; Mac OS X 10.6.8; de_DE)
 text/..
wAPI/1.1 (Bot: NoomBot Operator: Noommos Contact: mail address )
 application/vnd.php.serialized
gsa-crawler test (Enterprise; T3-N9HAEEX39WSGH; mail address )
 text/..
 -
GNAA-bot
 text/..
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
 text/..
Empedia Bot
 text/..
 -
Mozilla 5.0 (Apibot 0.30b5)
 application/vnd.php.serialized
WikiBot/0.1
 text/..
 image/..
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
 application/json
Metabot 0.1
 text/..
FactualWikiBot/0.1
 text/..
 application/json
Xaldon WebSpider 2.7.b8
 text/..
Mozilla/5.0 (bot; user:amalthea; en-us)
 -
 text/..
DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
athinchay-crawler/nutch-1.2 (Web crawler by athinchay; www.athinchay.com; mail address )
 text/..
Handelabra WikiBot
 application/vnd.php.serialized
 text/..
MySpider/Nutch-1.4
 text/..
 image/..
Wiki.java 0.26 (OctraBot 1.2)
 text/..
 -
 application/x-www-form-urlencoded
wikbotlite/1.60 CFNetwork/609 Darwin/13.0.0
 image/..
 application/json
 text/..
360Spider
 text/..
 -
 image/..
Mozilla/5.0 QunarBot/1.0
 text/..
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 367 subscribers; feed-id=3852576738117026533)
 application/xml
 -
PopScreenBot
 text/..
 image/..
python-wikitools/1.2 (User:LaraBot)
 application/json
Woozie! Crawl
 text/..
wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0
 text/..
 image/..
 application/json
scrapybot/1.0
 text/..
percbotspider mail address
 text/..
 image/..
EasouSpider
 text/..
 image/..
BibBot/0.9 (urshofer.ch)
 text/..
Nutch/Nutch-2.0 (Nutch Crawler)
 text/..
 -
lssbot
 text/..
 application/ogg
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.1.0/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
EarwigBot/0.2.dev.git4ff7612a (Python/2.7.3; https://github.com/earwig/earwigbot; mail address )
 application/json
wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0
 application/json
 image/..
 text/..
25737.37total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Tue, Oct 9, 2012 12:51
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers