Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Jun 2013 - 30 Jun 2013 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

Warning: all recent Wikimedia traffic analysis reports have been generated from old scripts.

The scripts are orphaned, and have not been maintained for at least 6 months. Many bugs are considerably older.
Known Bugzilla bugs: 46190, 46191, 46195, 46201, 46205, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 46289

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 117,484,200 page requests (mime type text/html only!) per day are considered crawler requests, out of 618,673,170 external requests, which is 19.0%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6022) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6024) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6029) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6031) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6028) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: s~libmuteki2) - -
 www.google.com/bot.htmlNONE/wikipedia- Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6030) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6026) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6025) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/coop/cse/creftext/..PageFetcher-Google-CoOp;((url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..Java/1.6.0_27 AppEngine-Google; (url; appid: s~1989-0723-6023) - -
 code.google.com/p/gwtwiki/text/..JavaWikipediaAPI/3.0 url - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) ja,en-us;q=0.7,en;q=0.3 -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) de -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AlexaToolbar/alxg-3.1 AppEngine-Google; (url; appid: s~chauslj) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~italiatiki) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 KHTML Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (url; appid: s~feedly-social) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kauflog) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki) - -
 docs.google.com-Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~doatresizer) - -
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - -
 code.google.com/appengine-Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.22 KHTML Chrome/25.0.1364.155 Safari/537.22 AppEngine-Google; (url; appid: s~cetlegoagent) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9) - -
 code.google.com/appenginetext/..Acre/dev/53:904 factchecker.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0 AppEngine-Google; (url; appid: s~cwek-gate1) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleDocs GoogleApps; script; url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AlexaToolbar/alxg-3.1 AppEngine-Google; (url; appid: s~chauslj) zh-CN,zh;q=0.8 -
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: bereturning) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: appcodes) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: howtodoita) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) SitemapProbe - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~scroll-down-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy6) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: cgiproxy5) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxy5070) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxy6009) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: cgiproxy6) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url) -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en, * -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en, *
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en, *
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8 -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonGoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.html-GoogleBot/2.1 (url) en-us,en;q=0.5
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexImages/3.0; url) -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) -
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.comtext/..Mozilla/5.0 (YahooYSMcm/3.0.0; url) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; ) PHP/5.4 - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/text/..WikipediaApp_svc/1.5.13 (url; mail address ) com.orange.mmp/1.7.0 - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/4.0; url) -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/application/xmlAhrefsBot.Feeds v0.1; url - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmapplication/jsonMozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible;Sosospider/2.0;url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmimage/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmapplication/xmlMozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/.."Mozilla/5.0 (compatible; Sosospider/2.0; url)" zh-cn,zh-hk,zh-tw,en-us -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17 - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.6.0_41; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_17; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-25-generic; java 1.7.0_21; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-26-generic; java 1.7.0_21; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.3; java 1.6.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.9-gentoo; java 1.7.0_21; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows 8 6.2; java 1.7.0_17; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.4; java 1.6.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.35-minimal; java 1.7.0_10; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; i386 Linux 3.2.0-48-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-48-generic; java 1.8.0-ea; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-1.29-default; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-23-generic; java 1.6.0_27; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.4; java 1.6.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/cs) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-41-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-47-server; java 1.6.0_26; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-22-generic; java 1.7.0_21; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.8.0-22-generic; java 1.7.0_21; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/es) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-48-generic; java 1.7.0_21; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-48-generic; java 1.8.0-ea; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-45-generic; java 1.8.0-ea; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.13-xxxx-grs-ipv6-64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; i386 Linux 2.6.32-042stab068.8; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_35; Europe/de) url - -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-17-generic; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-gentoo-r1; java 1.7.0_21; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.8.0-sabayon; java 1.7.0_21; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/cs) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-sabayon; java 1.7.0_21; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; amd64 Linux 3.2.0-48-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.9.6-1-ARCH; java 1.7.0_40; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.8.0-ea; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.5; java 1.6.0_45; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-1.11-desktop; java 1.7.0_21; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-042stab072.10; java 1.7.0_19; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.8.0-25-generic; java 1.7.0_21; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/es) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_04; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_21; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.7.10-1.11-desktop; java 1.7.0_21; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_45; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-39-generic; java 1.6.0_27; Europe/lv) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.8.0-23-generic; java 1.7.0_21; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-47-server; java 1.6.0_26; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.0-6.dmz.1-liquorix-amd64; java 1.6.0_24; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; i386 Linux 3.9.6-1-ARCH; java 1.7.0_40; Europe/en) url en-us,en;q=0.5 -
sblog
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/-Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/application/jsonSeznamBot/3.0 (url) - -
wordpress
 tsjok45.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 josefboberg.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 robertlindsay.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 thesandymonocle.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 raymondpronk.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 tsjok45.wordpress.comtext/..WordPress/3.6-beta4; url - -
 robertlindsay.wordpress.comtext/..WordPress/3.6-beta4; url - -
 biblecontroversiesdotcom.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 biblecontroversiesdotcom.wordpress.comtext/..WordPress/3.6-beta4-24523; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 asyafaat.wordpress.comtext/..WordPress/3.6-beta4; url - -
 escogitur.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 ramanan50.wordpress.comtext/..WordPress/3.6-beta4-24523; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 terrytao.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
 bharatabharati.wordpress.comtext/..WordPress/3.6-beta3-24375; url - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/-Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
 help.coccoc.com/text/..coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
wikipedia
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation-WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.20.0 url - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 nl.wikipedia.org/wiki/Overleg_gebruiker:MrBlueSkyapplication/jsonMediaWiki::API/0.29 (url) - -
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationapplication/x-www-form-urlencodedWPCleaner (url) - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
73
 178.63.73.19-Article Level Metrics - url -
 178.63.73.19application/jsonArticle Level Metrics - url - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -
 www.majestic12.co.uk/bot.php?-Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) -
 www.majestic12.co.uk/bot.php?-Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
 www.archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 wiki.toolserver.org/view/GeoHack-Geohack (url) -
 toolserver.org/~dispenser/-CacheThumbs/1.2 (url) -
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/application/x-www-form-urlencodedDispensersTools (url) -
 toolserver.org/~dispenser/-DispensersTools (url) -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
pchome
 www.pchome.com.tw/pchomebot.htmtext/..Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
 www.pchome.com.tw/pchomebot.htm-Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article Level Metrics - url - -
 alm.plos.org-PLOS Article Level Metrics - url -
seoratio
 www.seoratio.de/zyklop-crawler/text/..Mozilla/5.0 (compatible; zyklop; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) - -
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
acoon
 www.acoon.de/robot.asptext/..Mozilla/5.0 (compatible; AcoonBot/4.11.1; url) - -
 www.acoon.de/robot.asptext/..Mozilla/5.0 (compatible; AcoonBot/4.11.1; url) de-de,de,en-us,en -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
 www.gnip.com/text/..10.196.6.45 UnwindFetchor/1.0 (url) - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
sourceforge
 linkchecker.sourceforge.net/text/..LinkChecker/7.4 (url) - -
 linkchecker.sourceforge.net/image/..LinkChecker/7.4 (url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
wikidict
 www.wikidict.detext/..url - -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.org-Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
scrapy
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
 scrapy.org-Scrapy/0.16.5 (url) en -
 scrapy.orgtext/..Scrapy/0.17.0 (url) en -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
 www.parsijoo.ir/image/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
 support.paper.li/entries/20023257-what-is-paper-li-Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
mediawiki
 www.mediawiki.org/-MediaWiki OAI Harvester 0.2 (url) -
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ar -
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
 labs.topsy.com/butterfly/-Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
holasoyramon
 www.holasoyramon.com/blogtext/..WordPress/3.5.1; url - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
wikimedia
 commons.wikimedia.org/wiki/User:Thumbnails_Check_Botimage/..Thumbnails_Check_Bot/0.1 (url; beta) - -
 tools.wikimedia.de/~daniel/-WikiSense (url) -
 meta.wikimedia.org/wiki/User:Tietewtext/..Cheebot/0.5.7 (url) - -
 tools.wikimedia.de/~daniel/text/..WikiSense (url) - -
 tools.wikimedia.de/~para/GeoCommons/-url -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url) - -
aramabeta
 www.aramabeta.comtext/..betasearch/Nutch-2.1 (Academic Beta Search; url; mail address ) - -
 www.aramabeta.comtext/..betasearch/Nutch-2.1 (Academic Beta Search; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/-Drupal (url) -
profound
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
moviecus
 www.moviecus.com/botcontactinfo.php-moviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
automoveisnaweb
 automoveisnaweb.com.br/wp2text/..WordPress/3.5.1; url - -
 automoveisnaweb.com.br/wordpresstext/..WordPress/3.5.1; url - -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
webmeup
 webmeup.com/crawler.htmltext/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
news
 www.news.net/text/..News.Net/1.1 (url; mail address ) BasedOnRawPHP - -
 www.news.net/application/jsonNews.Net/1.1 (url; mail address ) BasedOnRawPHP - -
vk
 vk.com/dev/Share-Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) - -
example
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.com/MyCoolTool/application/jsonMyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
 example.com/MyCoolTool/text/..MyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
yunyun
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Bee-Bumble Bee/1.0 (WikiApiary; url) -
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) - -
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 wummel.github.com/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.4; url) - -
 github.com/shuber/curltext/..Curl/PHP 5.3.22 (url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
pingdom
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
 superfeedr.com-Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
vermagerd
 www.vermagerd.be/wptext/..WordPress/3.4.2; url - -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.1 (url) - -
 www.yougorhymes.com/site/rhyme-bot-RhymeBot/0.1 (url) -
weborking
 weborking.comtext/..Weborking(url) - -
 weborking.comapplication/jsonWeborking(url) - -
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
trendiction
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
xbmc
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
zeerch
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; ZEERCHBOT; url) - -
gumgum
 gumgum.comtext/..url; mail address - -
ac
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - -
 cse.iitkgp.ac.in/~rprtext/..Dr Rajendra/Nutch-0.9 (Dr Rajendra; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckbot.html-DuckDuckBot/1.1; (url) - -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url) - -
 enwp.org/User:Hellknowz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET) - -
memidex
 www.memidex.com/_bottext/..Mozilla/5.0 (compatible; Memibot/1.0; url ) en -
http:
 http://emilkirkegaard.dk/xkcdwiki/text/..url - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
onemusicapi
 www.onemusicapi.comtext/..onemusicapi.com/beta url - -
fueto
 fueto.comapplication/jsonFueto (url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
kyndane
 www.kyndane.com/bot.htmltext/..mozilla/5.0 (compatible; kynbot/1.0; url; IN TESTING) - -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
readability
 readability.com/about/text/..Readability/36a234 - url - -
 readability.com/about/text/..Readability/243856 - url - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.96.3; url) - -
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.96.2; url) - -
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url) en-us,en;q=0.5 -
sf
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url) en-us,en;q=0.5 -
 magpierss.sf.nettext/..MagpieRSS/0.7x (url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url) en-us,en;q=0.5 -
omgili
 omgili.comtext/..omgilibot/0.4 url en-us,en;q=0.5 -
wandex
 wandex.nettext/..Mozilla/5.0 (compatible; World Wide Web Wanderer (Wandex Bot)/1.4; url) - -
homeadvisorcolorado
watchmouse
tt-rss
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
igrec
 www.igrec.ca/projectstext/..Wikitionary Text Parser 0.2 (url) - -
zapbot
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url) - -
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url) - -
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url) - -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
rockmelt
 rockmelt.comtext/..RockmeltEmbedder (url; mail address ) - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
potaru
 potaru.com/robo.htmltext/..Mozilla/5.0 (compatible; Robo/1.0b; url)/Nutch-1.2 en-us,en-gb,en;q=0.7,*;q=0.3 -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
seomoz
 www.seomoz.org/dp/rogerbottext/..rogerbot/1.0 (url, mail address ) - -
uni-potsdam
 www.hpi.uni-potsdam.de/meinel/forschung/web_30/blog_intelligence.htmltext/..HPI-BI-Crawler/0.1(url)/Nutch-2.0-dev - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
publicknowledgeproject
 alm.publicknowledgeproject.orgapplication/jsonArticle Level Metrics - url - -
hvacrepairmedic
 denver.hvacrepairmedic.comtext/..WordPress/3.5.1; url - -
 aurora.hvacrepairmedic.comtext/..WordPress/3.5.1; url - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
backgroundswitcher
harvard
 cyber.law.harvard.edutext/..mediawords bot (url) - -
chickyrun
 chickyrun.tk/text/..ChickyBot/1.1 (url; mail address ) - -
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) - -
apercite
 www.apercite.fr/robot/index.html-Mozilla/5.0 (compatible; Apercite; url) en-us,en;q=0.5
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url) en-us,en;q=0.5 -
nvdev
 qa.nvdev.comtext/..Netvibes (url) - -
dasdonkey
 www.dasdonkey.comtext/..Mozilla/5.0 (compatible; DonkeyBot/0.2; url) - -
 www.dasdonkey.comtext/..Mozilla/5.0 (compatible; DonkeyBot/0.1; url) - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -
151007.959999973total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
wikiwix-bot-3.0 - -
 text/..
 -
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 text/..
 image/..
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
LinkParser/2.0 - -
 text/..
PythonWikipediaBot/1.0 - -
 application/json
 application/xml
 text/..
 -
tigerbot - -
 -
 application/json
 text/..
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
 application/json
 audio/webm
PythonWikipediaBot/1.0 -
 -
 application/x-www-form-urlencoded
Peachy MediaWiki Bot API Version 2.0 (alpha) - -
 application/vnd.php.serialized
 text/..
redirect/r11617 Pywikipediabot/2.0 - -
 application/json
YisouSpider - -
 text/..
 -
 image/..
redirect/r11694 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
ClueBot/1.1 - -
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
spider - -
 text/..
 application/vnd.php.serialized
 application/json
 -
 image/..
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
 text/..
redirect/r11686 Pywikipediabot/2.0 - -
 application/json
redirect/r11675 Pywikipediabot/2.0 - -
 application/json
My Nutch Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
redirect/r11650 Pywikipediabot/2.0 - -
 application/json
redirect/r11651 Pywikipediabot/2.0 - -
 application/json
redirect/r11601 Pywikipediabot/2.0 - -
 application/json
redirect/r11620 Pywikipediabot/2.0 - -
 application/json
ClueBot/2.0 - -
 application/vnd.php.serialized
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-interwiki.py/r11096 Pywikipediabot/1.0 - -
 application/xml
 application/json
Sogou web spider/4.0 zh-cn -
 text/..
 -
Peachy MediaWiki Bot API Version 2.0 (alpha 3) - -
 application/vnd.php.serialized
 text/..
WikiCatResearchBot ( mail address ) - -
 text/..
DigitalsmithsBot - -
 text/..
Mozilla 5.0 (Apibot 0.32) - -
 application/vnd.php.serialized
maj_articles_recents/r11650 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 image/..
 video/webm
 audio/webm
 unknown/unknown
 application/vnd.php.serialized
redirect/r11695 Pywikipediabot/2.0 - -
 application/json
redirect/r11689 Pywikipediabot/2.0 - -
 application/json
redirect/r11699 Pywikipediabot/2.0 - -
 application/json
redirect/r11657 Pywikipediabot/2.0 - -
 application/json
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
UAA-UAA.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 text/..
redirect/r11667 Pywikipediabot/2.0 - -
 application/json
redirect/r11639 Pywikipediabot/2.0 - -
 application/json
redirect/r11631 Pywikipediabot/2.0 - -
 application/json
redirect/r11646 Pywikipediabot/2.0 - -
 application/json
redirect/r11700 Pywikipediabot/2.0 - -
 application/json
BLEXBot - -
 text/..
 -
 application/json
fa-welcome.py/r11054 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
fa-welcome.py/r11054 Pywikipediabot/1.0 - -
 application/json
redirect/r11634 Pywikipediabot/2.0 - -
 application/json
MediaWiki::Bot/3.2.6 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 -
new2-zzdi1.py/r11452 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
new2-zzdi1.py/r11452 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 application/xml
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pywikipedia3-MakeScoreTable_test.py/r11448 Pywikipediabot/1.0 - -
 application/json
pywikipedia-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
CCBot/2.0 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
Tawbot (public svn release; plwiki) - -
 text/..
redirect/r11637 Pywikipediabot/2.0 - -
 application/json
pywikipedia-featured.py/r11675 Pywikipediabot/1.0 - -
 application/json
 application/xml
minicrawler_buaa - -
 text/..
GoogleBot-Image/1.0 -
 -
 image/..
 text/..
moje-wymowa.py/r11483 Pywikipediabot/1.0 -
 -
python-wikitools/1.2 (User:Irclogbot) -
 application/x-www-form-urlencoded
moje-wymowa.py/r11483 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
male/r11653 Pywikipediabot/2.0 - -
 application/json
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
pywikipedia-redirect.py/r11692 Pywikipediabot/1.0 - -
 application/json
GermCrawler - -
 application/json
 text/..
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
p-welcome-d-bn-core.py/r10872 Pywikipediabot/1.0 -
 -
Desktop-test.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
p-welcome-d-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11641 Pywikipediabot/1.0 - -
 application/json
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
matilda/r11686 Pywikipediabot/2.0 - -
 application/json
pywikipediabot_old-wikidata_new_global.py/r11364 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-featured.py/r11635 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywikipedia-radeh.py/r11655 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11618 Pywikipediabot/1.0 - -
 application/json
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
Test Webbot - -
 text/..
moje-porzucone.py/r11483 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
My Nutch Spider/Nutch-2.1 - -
 text/..
 application/ogg
Wikibot/2.1.1 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 text/..
startcmd/r11569 Pywikipediabot/2.0 - -
 application/json
wd_import/r11653 Pywikipediabot/2.0 - -
 application/json
moje-porzucone.py/r11483 Pywikipediabot/1.0 -
 -
botjagwar-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
pyreza-radeh.py/r11593 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11692 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
pyreza-radeh.py/r11593 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.43; ) - -
 text/..
pywikipedia-redirect.py/r11641 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
milog_bot/1.0 ( mail address ) - -
 text/..
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
ko_short/r11636 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzinter.py/r11593 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
pywikipedia-redirect.py/r11655 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; SearchBot) - -
 text/..
pywikipedia-importwd.py/r11618 Pywikipediabot/1.0 - -
 application/json
 -
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 -
MediaWiki::Bot/5.005006 - -
 application/json
pywikipedia-featured.py/r11597 Pywikipediabot/1.0 -
 -
bot: fr-anal - -
 application/json
maj_articles_recents/r11601 Pywikipediabot/2.0 - -
 application/json
pywikipedia-featured.py/r11597 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11618 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
VWBot - -
 application/json
 text/..
wd_universal_import/r11473 Pywikipediabot/2.0 - -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/x-www-form-urlencoded
 application/json
maj_articles_recents/r11601 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
Wikirage.Com Statistics Bot - -
 text/..
Semantix Bot 0.1 - -
 text/..
pywikipedia-featured.py/r11534 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikipedia-featured.py/r11618 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzinter.py/r11655 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
 application/xml
Python27-pythonw.exe/r11618 Pywikipediabot/1.0 - -
 application/json
WikidataBot framework - -
 application/json
 text/..
BotReversor-BotReversor.py/r11604 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
wikidata-sitelinks.py/r11618 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
FAST Search Web Crawler 14.0.0325.0000 - -
 text/..
 -
 application/ogg
 application/pdf
BotReversor-BotReversor.py/r11604 Pywikipediabot/1.0 - -
 application/json
FAST Enterprise Crawler 6 used by MS ( mail address ) - -
 text/..
 -
php wikibot classes -
 -
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
Wikidata-cbdb.py/r11202 Pywikipediabot/1.0 - -
 application/json
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
COIBot/1.00 - -
 text/..
WikiBot/0.1 - -
 text/..
 image/..
 application/xml
wdsex/r11276 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
HersfoldIRCBot version 1.8 - -
 text/..
 -
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
www.monit24.pl-m24Bot/4.1- - -
 image/..
 -
 text/..
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia2-redirect.py/r11692 Pywikipediabot/1.0 - -
 application/json
Simple Page Crawler. Contact: mail address en-us -
 text/..
Python27-pythonw.exe/r11571 Pywikipediabot/1.0 - -
 application/json
logpatrol/r11564 Pywikipediabot/2.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 -
 -
AnomieBOT 1.0 (PUICloser; see [[User:AnomieBOT]]) - -
 application/json
FTRF: Friendly robot/1.3 -
 -
 text/..
pywikipedia-fr.wikt.RemiseEnForme.py/r10175 Pywikipediabot/1.0 - -
 application/json
 application/xml
FTRF: Friendly robot/1.3 - -
 text/..
 -
aleph-avbot.py/r11368 Pywikipediabot/1.0 - -
 application/json
 text/..
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
Twitterbot/1.0 - -
 text/..
 image/..
 -
 application/pdf
 application/ogg
trunk-checkimages.py/r11618 Pywikipediabot/1.0 - -
 application/json
HTMLParser/2.0 - -
 text/..
 -
 image/..
COIBot/2.0 - -
 text/..
Bot: mail address - -
 text/..
pywikipedia-fr.wikt.RemiseEnForme.py/r11692 Pywikipediabot/1.0 - -
 application/json
aleph-avbot.py/r11368 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
pywikirish-welcome.py/r11215 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Wiki.java 0.27 r137 (OctraBot 2.6) - -
 -
 text/..
 application/json
 application/x-www-form-urlencoded
DeletionBot-deletion.py/r11525 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikirish-welcome.py/r11215 Pywikipediabot/1.0 - -
 application/json
DeletionBot-deletion.py/r11525 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-featured.py/r11620 Pywikipediabot/1.0 - -
 application/json
 application/xml
FAST Enterprise Crawler 6 used by Bygglo / Svensk byggtjanst - -
 text/..
 -
pywikipedia-redirect.py/r11674 Pywikipediabot/1.0 - -
 application/json
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
 application/x-www-form-urlencoded
Epywikipedia-welcome.py/r11181 Pywikipediabot/1.0 - -
 application/json
WPBot 1.0 - -
 text/..
maj_articles_recents/r11620 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
merge/r11473 Pywikipediabot/2.0 - -
 application/json
360spider-image - -
 image/..
maj_articles_recents/r11700 Pywikipediabot/2.0 - -
 application/json
pywikipedia4-commonscat.py/r11694 Pywikipediabot/1.0 - -
 application/json
 application/xml
maj_articles_recents/r11620 Pywikipediabot/2.0 - -
 application/json
pywikipedia2-redirect.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11593 Pywikipediabot/1.0 - -
 application/json
pywikipedia-category_redirect.bak.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-28main.build89.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
pywikipedia-redirect.py/r11655 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r11690 Pywikipediabot/1.0 - -
 application/json
Addbot Wikimedia Bot - -
 text/..
 application/vnd.php.serialized
 -
HRoestBot, de-wikipedia using pywikipedia framework -
 application/x-www-form-urlencoded
 -
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) - -
 application/json
pywikipedia3-MakeScoreTable_test2.py/r11448 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
rg/r11515 Pywikipediabot/2.0 - -
 application/json
iw2data/r11653 Pywikipediabot/2.0 - -
 application/json
pywiki3-wikidata_coord_bui2.py/r11650 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11633 Pywikipediabot/1.0 - -
 application/json
Py-new_interwiki_sk.py/r11596 Pywikipediabot/1.0 - -
 application/json
 application/xml
JavaCrawler/1.1 - -
 text/..
 -
 image/..
SurakWare MediaWiki Bot/1.0 - -
 text/..
 application/xml
Robot911 - -
 text/..
 -
 image/..
 application/xml
 application/opensearchdescription+xml
 application/rsd+xml
pywikipedia-featured.py/r11681 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11674 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11674 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
maj_articles_recents/r11639 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
redirect/r11596 Pywikipediabot/2.0 - -
 application/json
matilda/r11675 Pywikipediabot/2.0 - -
 application/json
www.monit24.pl-m24Bot/4.1- -
 -
pywikipedia-touch.py/r11530 Pywikipediabot/1.0 - -
 application/json
 application/xml
maj_articles_recents/r11639 Pywikipediabot/2.0 - -
 application/json
pywikipedia-featured.py/r11681 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-wikidata_setproperty_NL_frominfobox_gemeentelink.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
TicketsBot/0.1 - -
 text/..
pywikipedia-redirect.py/r11633 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r11681 Pywikipediabot/1.0 - -
 application/json
Projet IW-PIW2.py/r10533 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-welcome-custom.py/r11354 Pywikipediabot/1.0 - -
 application/json
pywikipedia-28main.build89.py/r11399 Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-VM-auto-erl.py/r11602 Pywikipediabot/1.0 - -
 application/json
botjagwar-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
FAST Search Web Crawler 14.0.0325.0000 -
 -
 text/..
pywikipedia-wikidata_add_label_only.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
trunk-featured.py/r11645 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
 application/ogg
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) - -
 text/..
pywikipedia2-redirect.py/r11655 Pywikipediabot/1.0 - -
 application/json
trunk-featured.py/r11650 Pywikipediabot/1.0 - -
 application/json
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) -
 -
unknown, 2a00:4b00:13d:200::129e Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
pywikipedia-interwiki.py/r11618 Pywikipediabot/1.0 - -
 application/xml
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-zzlangcat.py/r11618 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-featured.py/r11666 Pywikipediabot/1.0 - -
 application/json
 application/xml
unknown, 2a00:4b00:13d:200::129c Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
XLinkBot/1.00 - -
 text/..
Webbull-Bot - -
 text/..
unknown, 2a00:4b00:13d:200::129a Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
pywikipedia2-redirect.py/r11618 Pywikipediabot/1.0 - -
 application/json
pywikipedia-category_redirect.py/r11610 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11655 Pywikipediabot/1.0 - -
 application/xml
 application/json
unknown, 2a00:4b00:13d:200::129b Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
unknown, 2a00:4b00:13d:200::129f Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
hercule-commonscat.py/r11618 Pywikipediabot/1.0 - -
 application/json
Empedia Bot - -
 text/..
new_interwiki/r11473 Pywikipediabot/2.0 - -
 application/json
trunk-featured.py/r11634 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-radeh.py/r11593 Pywikipediabot/1.0 - -
 application/json
enwiki-sandheaderbot.py/r11533 Pywikipediabot/1.0 - -
 application/json
 -
hercule-commonscat.py/r11618 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
botjagwar-crawlrandom.py/r11215 Pywikipediabot/1.0 - -
 application/json
2217286/r11657 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11687 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
unknown, 2a00:4b00:13d:200::129d Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
2217286/r11657 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r11624 Pywikipediabot/1.0 - -
 application/json
2114381/r11563 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
hercule-commonscat.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-pwb.py/r11615 Pywikipediabot/1.0 - -
 application/json
 image/..
 application/xml
 text/..
hercule-commonscat.py/r11692 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Python27-pythonw.exe/r10533 Pywikipediabot/1.0 - -
 application/json
 application/xml
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11687 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-pwb.py/r11615 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-iw2data.py/r11618 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikt-interwiki.py/r11597 Pywikipediabot/1.0 - -
 application/xml
 application/json
pywikipedia-radeh.py/r11231 Pywikipediabot/1.0 -
 -
rg/r11515 Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
AdMedia bot - -
 text/..
 -
request-daemon/r11515 Pywikipediabot/2.0 - -
 application/json
2114381/r11563 Pywikipediabot/2.0 - -
 application/json
pywikipedia-basicdisambrefslist.py/r11597 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
PhronesisSpider - -
 text/..
commons-imagerecat.py/r11028 Pywikipediabot/1.0 - -
 application/json
 -
SchoolReviewNetworkWikiBot - -
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pywikipedia-redirect.py/r11629 Pywikipediabot/1.0 - -
 application/json
theWxitBot/0.1 - -
 application/json
 image/..
pywikipedia-voverb.py/r11215 Pywikipediabot/1.0 - -
 application/json
Zing-BottaBot/2.0 - -
 text/..
pywikipedia-redirect.py/r11599 Pywikipediabot/1.0 - -
 application/json
trunk-featured.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
hercule-commonscat.py/r11641 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
TVersity Media Robot - -
 text/..
hercule-commonscat.py/r11692 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-basicdisambrefslist.py/r11597 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-17-progreso1911.py/r11618 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pyiw-touch.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
dtSearchSpider - -
 text/..
pywikt-interwiki.py/r11597 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-commons_redirect_rep.py/r11641 Pywikipediabot/1.0 - -
 application/json
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 -
 -
pywikipedia-basicsbz.py/r11597 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-basicsbz.py/r11597 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11690 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
SkimBot/1.0 (www.skimlinks.com mail address ) en-us,en;q=0.5 -
 text/..
 -
 image/..
Java bot - -
 application/json
User-Agent: Futuramabot for Hubot (https://github.com/github/hubot-scripts) - -
 text/..
 -
pywikipedia-vyzreplace.py/r11366 Pywikipediabot/1.0 - -
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
 -
pywikipedia-17-progreso1911.py/r11618 Pywikipediabot/1.0 - -
 application/json
User-Agent: Archerbot for Hubot (https://github.com/github/hubot-scripts) - -
 text/..
 -
pywikipedia-radeh.py/r11231 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.44; ) - -
 text/..
pywikipedia-redirect.py/r11681 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r11385 Pywikipediabot/1.0 - -
 application/json
 text/..
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
pywikipedia-redirect.py/r11385 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia2-redirect.py/r11674 Pywikipediabot/1.0 - -
 application/json
pywikipedia-17-progreso1911.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11624 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
Wikipediabot-wikidata_gndcheck.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
ReaperBot/2.0.0 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot> - -
 text/..
pywikilab-llista_mils_2013.py/r11604 Pywikipediabot/1.0 -
 -
pywikipedia-17-progreso1911.py/r11641 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikilab-llista_mils_2013.py/r11604 Pywikipediabot/1.0 - -
 application/json
pywikipedia-category_redirect.py/r11624 Pywikipediabot/1.0 - -
 application/json
webbot - www.unnex.de - -
 text/..
pywikipedia-_move2wikidata_np_ua.py/r11508 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
 application/pdf
pywikipedia-category_redirect.bak.py/r11633 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 0.1beta - -
 application/vnd.php.serialized
Pywikipedia-new_interwiki.py/r11579 Pywikipediabot/1.0 - -
 application/json
pywikipedia-17-progreso1911.py/r11692 Pywikipediabot/1.0 -
 -
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1) - -
 application/json
trunk-checkimages.py/r11667 Pywikipediabot/1.0 - -
 application/json
wpscrawl/1.0 en -
 text/..
 -
 application/xml
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
Web Crawler - -
 text/..
pywikipedia2-redirect.py/r11690 Pywikipediabot/1.0 - -
 application/json
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
pywikipedia-interwiki.py/r11618 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 text/..
MediaWiki::Bot/3.005002 - -
 application/json
pywikipedia4-featured.py/r11694 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
opentask-opentasks.py/r10326 (pywikibot/__init__.py) Pywikipediabot/2.0 -
 application/x-www-form-urlencoded
Peachy MediaWiki Bot API Version 0.1beta -
 -
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 -
pywikipedia-category_redirect.py/r11633 Pywikipediabot/1.0 - -
 application/json
 application/xml
wikiscore-MakeScoreTable.py/r11549 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
CiooxBot - -
 text/..
python-wikitools/1.2 (User:LaraBot) -
 application/x-www-form-urlencoded
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1) -
 application/x-www-form-urlencoded
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
artists/r11515 Pywikipediabot/2.0 - -
 application/json
pywikipedia-17-progreso1911.py/r11692 Pywikipediabot/1.0 - -
 application/json
 application/xml
trunk-checkimages.py/r11650 Pywikipediabot/1.0 - -
 application/json
trunk-featured.py/r11635 Pywikipediabot/1.0 - -
 application/json
pywikipedia-importwd.py/r11597 Pywikipediabot/1.0 - -
 application/json
EarwigBot/0.2.dev.gitca7cbceb (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
commons-nulledit.py/r11618 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 text/..
pywikipedia-getpron.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_missinglabel_add.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
t9/r11564 Pywikipediabot/2.0 - -
 application/json
datetemplates/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
python-wikitools/1.2 (User:LaraBot) - -
 application/json
pywikipedianew-checkimages.py/r11550 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
matilda/r11667 Pywikipediabot/2.0 - -
 application/json
pywikipedia-featured.py/r9034 Pywikipediabot/1.0 - -
 application/json
subtemplates/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia2-redirect.py/r11624 Pywikipediabot/1.0 - -
 application/json
pywikipedia-17-progreso1911.py/r11655 Pywikipediabot/1.0 - -
 application/json
 application/xml
Math. Comp. PA Crawler - -
 text/..
pywikipedia-solve_disambiguation.py/r11597 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-wikidata_disamb.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
MediaWiki::Bot 3.1.5 - -
 application/json
pywikipedia-featured.py/r11657 Pywikipediabot/1.0 - -
 application/json
 application/xml
hercule-commonscat.py/r11674 Pywikipediabot/1.0 - -
 application/json
 application/xml
Math. Comp. PA Crawler -
 -
 text/..
(u'python-wikitools/1.1.1 (User:Cerabot)',) - -
 application/json
Phantom.js bot cs-CZ,en,* -
 image/..
 text/..
pywikipedia4-wikidata_labelallcats.py/r11694 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Bot 3.1.5 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.102 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
hercule-commonscat.py/r11624 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-pagefromfile.py/r11200 Pywikipediabot/1.0 - -
 application/json
hercule-commonscat.py/r11674 Pywikipediabot/1.0 -
 -
botpywi-analyse_article.py/r11508 Pywikipediabot/1.0 - -
 application/json
 application/xml
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) -
 -
pywikipedia-redirect.py/r11622 Pywikipediabot/1.0 - -
 application/json
('python-wikitools/1.1.1 (User:BernsteinBot)',) -
 application/x-www-form-urlencoded
Py-new_interwiki.py/r11596 Pywikipediabot/1.0 - -
 application/json
pywikipedia-archivebot.py/r11618 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-archivebot.py/r11618 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-archivebot.py/r11692 Pywikipediabot/1.0 - -
 application/json
pywikipedia2-redirect.py/r11681 Pywikipediabot/1.0 - -
 application/json
commons-substituter.py/r11436 Pywikipediabot/1.0 -
 -
EarwigBot/0.2.dev.gitca7cbceb (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) -
 application/x-www-form-urlencoded
pywikipedia-welcome2.py/r11641 Pywikipediabot/1.0 - -
 application/json
newser-wikinewser2.py/r11692 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11599 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-welcome2.py/r11692 Pywikipediabot/1.0 - -
 application/json
('python-wikitools/1.1.1 (User:BernsteinBot)',) - -
 application/json
gsa-crawler (Enterprise; GID-01422; jplastiras.com) - -
 text/..
 -
pywikipedia-welcome2.py/r11641 Pywikipediabot/1.0 -
 -
pywikipedia-solve_disambiguation.py/r11597 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
hercule-commonscat.py/r11624 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-wikidata_cleanlabel.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-archivebot.py/r11641 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
hercule-commonscat.py/r11690 Pywikipediabot/1.0 -
 -
Screaming Frog SEO Spider/2.11 - -
 text/..
 image/..
 -
bitlybot - -
 text/..
 image/..
 -
commons-substituter.py/r11436 Pywikipediabot/1.0 - -
 application/json
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
 text/..
newser-wikinewser2.py/r11692 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
pywikipedia-replace.py/r11522 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-welcome2.py/r11618 Pywikipediabot/1.0 -
 -
pywikipedia-randomretriever.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywikipedia-archivebot.py/r11641 Pywikipediabot/1.0 - -
 application/json
newser-wikinewser2.py/r11618 Pywikipediabot/1.0 - -
 application/json
pywikipedia-touch.py/r11601 Pywikipediabot/1.0 - -
 application/json
mrajedrez-category_redirect.py/r11321 Pywikipediabot/1.0 - -
 application/json
pywikipedia-welcome2.py/r11692 Pywikipediabot/1.0 -
 -
opentask-opentasks.py/r10326 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
py_framework-edit-liste.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11629 Pywikipediabot/1.0 -
 -
pywiki3-wikidata_coord.py/r11650 Pywikipediabot/1.0 - -
 application/json
file_has_rationale_yes/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11604 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata9nwd_nl.py/r11364 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
pywikipedia-17-progreso1911.py/r11655 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Feedstripes Bot v1.0 - -
 text/..
 -
 application/xml
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot en-us,en;q=0.5 -
 text/..
gsa-crawler (Enterprise; T3-GMDVWEKT9WSQZ; mail address ) - -
 text/..
 image/..
pywikipedia-redirect.py/r11603 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
newser-wikinewser2.py/r11618 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
33872.64total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Tue, Jul 9, 2013 11:35
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers