Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Jan 2014 - 31 Jan 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 118,381,100 page requests (mime type text/html only!) per day are considered crawler requests, out of 595,512,450 external requests, which is 19.9%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..37.58.85.180 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)Accept: text/html,application/xhtmlxml,application/xml;image/png,image/jpeg,image/*;q=0.9,*/*;q=0.8 en-us;q=0.7, en;q=0.3 -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~doingfly) - -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tinyproxys) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..Acre/dev/53:927 proxy.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ageryder) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~warawasetekure) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: easypox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~tinyproxys) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appengineimage/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 AppEngine-Google; (url; appid: s~revdonut-01) zh-cn,zh;q=0.7,en-us;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icebre4ker) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~freekaznet) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376eSafari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: minecraft6699) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beholdonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~dizykit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icecool1987) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/31.0.1650.63 Safari/537.36 AppEngine-Google; (url; appid: s~ffffnm) zh-CN,zh;q=0.8,zh-TW;q=0.6 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: randohnson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - -
 code.google.com/appenginetext/..Acre/dev/53:922 proxy.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lil05warrior) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ungoodthinkful) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 code.google.com/appenginetext/..Acre/dev/59:906 ubiquity.freebaseapps.com AppEngine-Google; (url; appid: s~freebaseapps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: javahavens) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jttm-server-prox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hoptheborder) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ownsterserver) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 KHTML Chrome/31.0.1650.63 Safari/537.36 AppEngine-Google; (url; appid: s~nsftcn) zh-CN,zh;q=0.8 -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kylesumrallproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ikaryse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: anonchurro) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thegatetoweb) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..37.58.85.180 msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..37.58.85.180 Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/rsd+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh4) en-US -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: jpg8jpg) en-US -
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.2 - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~hhff2h) en-US -
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..37.58.85.180 Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmlapplication/pdfGoogleBot/2.1 (url) - -
 www.google.com/bot.htmlimage/..GoogleBot/2.1 (url) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/text/..WikipediaApp_svc/1.5.13 (url; mail address ) com.orange.mmp/1.7.0 - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..37.58.85.180 Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
 ahrefs.com/application/xmlAhrefsBot.Feeds v0.1; url - -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.103 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.105 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.107 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.102 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.101 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.100 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.104 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.108 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.99 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..98.137.206.106 Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-35-generic; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-35-generic; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-12-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-042stab084.3; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.7; java 1.7.0_40; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12-1-amd64; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-1.16-desktop; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-1.24-desktop; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_03; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.41-042stab081.5; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11-2-amd64; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-45-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_27; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.7-2-ARCH; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12-1-amd64; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.7; java 1.7.0_51; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.9-1-ARCH; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_40; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; i386 Linux 3.11.0-15-generic; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-042stab083.2; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-58-generic; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.6-1-ARCH; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-15-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_35; Europe/de) url - -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Vista 6.1; java 1.6.0_13; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.0-031200-generic; java 1.7.0_25; Asia/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0-gentoo-r1; java 1.6.0_27; GMT/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-26-pve; java 1.6.0_27; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-26-pve; java 1.7.0_25; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-12-generic; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_45; GMT-03:00/pt) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.6-1-CHAKRA; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmlimage/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.6-1-ck; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.11-2-686-pae; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-35-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-57-generic; java 1.6.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.9.9; java 1.7.0_09; Hongkong/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmlimage/..yacybot (freeworld/global; amd64 Linux 3.8.0-35-generic; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-15-generic; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.4-1-ARCH; java 1.7.0_45; GMT/en) url en-us,en;q=0.5 -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
http:
 http://alm.labs.crossref.orgapplication/jsonArticle Level Metrics - url -
 http://alm.example.orgapplication/jsonArticle Level Metrics - url - -
 http://www.goguide-directory.com.au/application/vnd.php.serializedTesting for url - -
 http://www.goguide-directory.com.au/application/vnd.php.serializedurl - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/0.5.2 (url) PHP/5.3.13p1 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.28 - -
sblog
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) - -
 fulltext.sblog.cz/application/jsonSeznamBot/3.0 (url) - -
wordpress
 wildanrenaldi.wordpress.comtext/..WordPress/3.9-alpha; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.8.1; url - -
 wikimediaukraine.wordpress.comtext/..WordPress/3.8.1; url - -
 whatownsme.wordpress.comtext/..WordPress/3.9-alpha; url - -
 peripluscd.wordpress.comtext/..WordPress/3.9-alpha; url - -
 02varvara.wordpress.comtext/..WordPress/3.9-alpha; url - -
 menschlein.wordpress.comtext/..WordPress/3.9-alpha; url - -
 y35pm.wordpress.comtext/..WordPress/3.9-alpha; url - -
 anarchyisorder.wordpress.comtext/..WordPress/3.9-alpha; url - -
 josefboberg.wordpress.comtext/..WordPress/3.9-alpha; url - -
 generalfeldmarschall1staatpreussen.wordpress.comtext/..WordPress/3.9-alpha; url - -
 tsjok45.wordpress.comtext/..WordPress/3.9-alpha; url - -
 y35pm.wordpress.comtext/..WordPress/3.8.1; url - -
 manofroma.wordpress.comtext/..WordPress/3.9-alpha; url - -
 menschlein.wordpress.comtext/..WordPress/3.8.1; url - -
 dshenai.wordpress.comtext/..WordPress/3.9-alpha; url - -
 dearkitty1.wordpress.comtext/..WordPress/3.9-alpha; url - -
 ladansedusinge.wordpress.comtext/..WordPress/3.9-alpha; url - -
 hrexach.wordpress.comtext/..WordPress/3.9-alpha; url - -
 halokue.wordpress.comtext/..WordPress/3.9-alpha; url - -
 tsjok45.wordpress.comtext/..WordPress/3.8.1; url - -
 thejackelscolumn.wordpress.comtext/..WordPress/3.9-alpha; url - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
 nodejs.orgapplication/jsonNodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
wikipedia
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 en.wikipedia.orgtext/..url - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) -
 www.majestic12.co.uk/bot.php?text/..(compatible; MJ12bot/v1.4.4; url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
flipboard
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
bin-co
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ru -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address es-es -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics - url - -
Anonymouse
 Anonymouse.org/text/..url (Unix) - -
 Anonymouse.org/image/..url (Unix) - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
mydevtest
 www.mydevtest.com/application/jsonMyTest/1.0 (url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 - -
localhost
 localhost/clumpit/bot.phptext/..Mozilla/5.0 (compatible; clumboot; url) - -
 localhost/bot.phptext/..Mozilla/5.0 (compatible; clumboot; url) - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.comtext/..FeedlyApp/1.0 (url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
websearchengine
 websearchengine.orgtext/..WSE/1.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
archive-it
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
wesee
 www.wesee.com/en/support/bot/image/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/en/support/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) -
 toolserver.org/~dispenser/text/..DispensersTools (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
automoveisnaweb
 automoveisnaweb.com.br/uk4text/..WordPress/3.6.1; url - -
 automoveisnaweb.com.br/uk3text/..WordPress/3.7.1; url - -
 automoveisnaweb.com.br/uk3text/..WordPress/3.8.1; url - -
 automoveisnaweb.com.br/uk4text/..WordPress/3.8; url - -
xbmc
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.2;WOW64;Win64;x64; url) - -
scrapy
 scrapy.orgtext/..Scrapy/0.20.2 (url) en -
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.22.0 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.21.0 (url) en -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo-bot; url; mail address ) - -
 www.parsijoo.ir/image/..Mozilla/5.0 (compatible; parsijoo-bot; url; mail address ) - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.97; url) - -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
irit
 www.irit.fr/melodibot/text/..melodibot (url) en -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
example
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.com/MyCoolTool/application/jsonMyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
 alm.example.orgapplication/jsonArticle-Level Metrics - url - -
 example.comapplication/jsonArticle-Level Metrics - url - -
 example.com/MyCoolTool/text/..MyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
wikidict
 www.wikidict.detext/..url - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
github
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.1 Ruby/2.0.0p353 (url) en-us,en;q=0.5 -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
 wummel.github.com/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.4; url) - -
seoprofiler
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0.6; url ) - -
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0.5; url ) - -
site
 www.site.com/application/jsonBot/2.0 (url) -
diffbot
 www.diffbot.comtext/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 KHTML Chrome/29.0.1547.66 Safari/537.36 crawlbot/1.0 (url) - -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
mapsway
 mapsway.com/text/..MapswayBot/1.0 (url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
 www.proximic.com/info/spider.phpimage/..Mozilla/5.0 (compatible; proximic; url) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
 www.netvibes.comapplication/jsonNetvibes (url) de-de -
 www.netvibes.comapplication/jsonNetvibes (url) fr-fr -
trendiction
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
holasoyramon
 www.holasoyramon.com/blogtext/..WordPress/3.5.1; url - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (Macintosh; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
nvdev
 qa.nvdev.comtext/..Netvibes (url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
yunyun
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmltext/..url en,hi -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
mystudent
 techatat001.mystudent.intext/..WordPress/3.4.2; url - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url;) Gecko/2008032620 - -
linkdex
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
botify
 botify.comtext/..Mozilla/5.0 (compatible; botify; url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
ac
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
gorgor
 gorgor.irtext/..hivaBot(url and mail address )/hivaBot-0.2 en-us,en-gb,en;q=0.7,*;q=0.3 -
qbqm
 www.qbqm.fr/text/..QBQM/1.0 (url) fr -
netnewswireapp
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
dtdns
 www.archibald-birseaufoys.dtdns.net/bot/index.htmltext/..Jicreux/Nutch-1.7_evolution_4 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.archibald-birseaufoys.dtdns.net/bot/index.htmltext/..Jicreux/Nutch-1.7_evolution_3 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
web
 www.web.nl/text/..web.nl spider/1.7; url - -
com?type=714647&fr=spigot-yhp-ch
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) - -
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/3.0 (url) - -
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
vote-often
 vote-often.comtext/..WordPress/3.5.1; url - -
profound
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - -
watchmouse
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
tt-rss
simplepie
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
igetfixed
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url) - -
QtWeb
rebelmouse
 rebelmouse.comimage/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.comtext/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
accelobot
 www.accelobot.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
gravity
 www.gravity.com/text/..Mozilla/5.0 (compatible; Gravitybot/1.0; url) - -
 www.gravity.com/image/..Mozilla/5.0 (compatible; Gravitybot/1.0; url) - -
mytest
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131104-0244 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131104-0244 url) - -
image2play
 image2play.com/image/..Mozilla/5.0 (compatible; Image2play/0.1; url) - -
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url) en-us -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.8 (url) - -
queryseeker
 queryseeker.com/bot.htmltext/..QuerySeekerSpider ( url ) - -
bigdatacorp
 ecommerce.bigdatacorp.com.br/faq.aspxtext/..Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; url) AppleWebKit/535.1 KHTML Chrome/13.0.782.107 Safari/535.1 pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4 -
weserv
 images.weserv.nl/image/..Mozilla/5.0 (compatible; ImageFetcher/5.5; url) -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.3 (url) -
scarecrow
 www.scarecrow.com/application/jsonScarecrow/1.0 (url; mail address ) - -
quantactics
 www.quantactics.com/text/..QTFIL/1.0 (url; mail address ) - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
gg
 gg.pl/text/..GG PeekBot 2.0 ( url http://info.gadu-gadu.pl/praca ) - -
 gg.pl/image/..GG PeekBot 2.0 ( url http://info.gadu-gadu.pl/praca ) - -
orfeed
 bath.orfeed.comtext/..WordPress/3.8; url - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
bowsio
 bowsio.infotext/..WordPress/3.8; url - -
fueto
 fueto.comapplication/jsonFueto (url) - -
154346.789999985total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 6) - -
 application/vnd.php.serialized
 text/..
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 application/pdf
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/pdf
 application/ogg
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
LinkParser/2.0 - -
 text/..
wikiwix-bot-3.0 - -
 text/..
tigerbot - -
 application/json
 text/..
 image/..
 application/ogg
pywikipedia-rad1.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
maj_articles_recents/r2607 Pywikipediabot/2.0 - -
 application/json
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
ClueBot/1.1 - -
 application/vnd.php.serialized
PythonWikipediaBot/1.0 - -
 application/json
 text/..
 application/xml
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
GoogleBot-Image/1.0 - -
 text/..
 image/..
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 image/..
 application/xml
 application/ogg
ClueBot/2.0 - -
 application/vnd.php.serialized
maj_articles_recents/r2588 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
pwb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; SputnikBot/2.2) ru,en;q=0.1,*;q=0.01 -
 text/..
pwb-klbot_wikdiata_name2.py/r7 Pywikipediabot/1.0 - -
 application/json
 text/..
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 application/ogg
 image/..
MediaWiki::Bot/5.005006 - -
 application/json
Warrior Search-Stat robot - Building trusted news provider index - -
 application/json
BOT/0.1 (BOT for JCE) - -
 text/..
DotNetWikiBot/2.101 (Unix 3.2.0.54; ) - -
 text/..
 application/xml
DigitalsmithsBot - -
 text/..
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
freebasemappingupload/r2403 Pywikipediabot/2.0 - -
 application/json
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
Semantix Bot 0.1 - -
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 text/..
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/json
 application/xml
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pwb/r2665 Pywikipediabot/2.0 - -
 application/json
MediaWiki::Bot/3.2.6 - -
 application/json
 text/..
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 text/..
MediaWiki::Bot/3.005002 - -
 application/json
compat-welcome.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
PythonWikipediaBot/1.0 -
 application/json
 text/..
 application/xml
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywiki-newar25.py/r10754 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/r2669 Pywikipediabot/2.0 - -
 application/json
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
 application/ogg
 application/json
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.54; ) - -
 text/..
 application/xml
qic2/r2549 Pywikipediabot/2.0 - -
 application/json
Addbot Wikimedia Bot - -
 text/..
milog_bot/1.0 ( mail address ) - -
 text/..
redirect/r2605 Pywikipediabot/2.0 - -
 application/json
redirect/r2669 Pywikipediabot/2.0 - -
 application/json
authority/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Lynx Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
redirect/r2638 Pywikipediabot/2.0 - -
 application/json
pywikipedia-radeh.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
fa-welcome.py/r11054 Pywikipediabot/1.0 -
 application/json
dikantenyvaovao_beta/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
WikidataBot framework - -
 application/json
maj_articles_recents/r2556 Pywikipediabot/2.0 - -
 application/json
pywikipedia-wikifyer.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-interwiki.py/r10761 Pywikipediabot/1.0 - -
 application/json
 application/xml
360spider-image - -
 image/..
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider; zh-CN -
 text/..
anagrama/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
spider - -
 text/..
 application/json
 application/vnd.php.serialized
 image/..
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
GermCrawler - -
 application/json
 text/..
 application/xml
DotNetWikiBot/3.0 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
JackBot-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 -
 application/json
 text/..
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/r10751 Pywikipediabot/1.0 - -
 application/json
 application/xml
Webbot - -
 text/..
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
 text/..
compat-checkimages.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
redirect/r2620 Pywikipediabot/2.0 - -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
web-crawler en-US,en;q=0.8 -
 text/..
romaine/r2620 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph661.py/r90 Pywikipediabot/1.0 - -
 application/json
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
 text/..
Program-Proofreading_Contest_Count.py/r11747 Pywikipediabot/1.0 -
 application/json
pwb-reportCatLoops.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
run_replace.6r.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
compat-featured.py/r10441 Pywikipediabot/1.0 - -
 application/json
 application/xml
wikiparser/1 CFNetwork/673.0.3 Darwin/13.0.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
redirect/r2616 Pywikipediabot/2.0 - -
 application/json
mail address - -
 application/json
 text/..
pywikibot-1newar25.py/r10797 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-1newar25.py/r10810 Pywikipediabot/1.0 - -
 application/json
 application/xml
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph664.py/r90 Pywikipediabot/1.0 - -
 application/json
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 en-us -
 image/..
 text/..
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 KHTML Chrome/24.0.1290.2 Safari/537.13 Lumibot/2.0 en-US,en;q=0.8 -
 image/..
 text/..
 application/json
pywikibot-1newar25.py/r10809 Pywikipediabot/1.0 - -
 application/json
COIBot/1.00 - -
 text/..
pwb/r2652 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
run_replace.6r.0/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipediabot-replace.py/r10589 Pywikipediabot/1.0 - -
 application/json
 application/xml
SentiBot mail address (compatible with GoogleBot) - -
 text/..
 application/xml
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
redirect/r2631 Pywikipediabot/2.0 - -
 application/json
run_replace.6r.1/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
pywiki-cfd.py/r10360 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-interwiki.py/r10250 Pywikipediabot/1.0 - -
 application/xml
 application/json
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
Biosgraphies Online Crawler (Curl/PHP) mail address - -
 application/json
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-newinterwiki-fa.py/r10334 Pywikipediabot/1.0 - -
 application/json
pwb/r2631 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph661de.py/r90 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10734 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-dave_sezioni_test1.py/r10718 Pywikipediabot/1.0 - -
 application/json
 application/xml
harvest_nltaxo/r2458 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.56; ) - -
 text/..
 application/xml
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
redirect/r2652 Pywikipediabot/2.0 - -
 application/json
pwb/r2471 Pywikipediabot/2.0 - -
 application/json
redirect/r2684 Pywikipediabot/2.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
DanmicholoBot - -
 application/json
pywikipedia-zzinter.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2626 Pywikipediabot/2.0 - -
 application/json
Bot-interwiki.py/r10581 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/3.10 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
iw_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
COIBot/2.0 - -
 text/..
pywikibot-1newar25.py/r10794 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-basictemplatedisamblist.py/r10765 Pywikipediabot/1.0 - -
 application/json
 application/xml
Yeobot/1.0 (Contact: mail address ) - -
 text/..
 application/xml
pywikibot-1newar25.py/r10811 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2623 Pywikipediabot/2.0 - -
 application/json
compat-basictemplatedisamblist.py/r10810 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
commonscat_to_wikidata/r2506 Pywikipediabot/2.0 - -
 application/json
pywikibot-1newar25.py/r10805 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-zzporb-main.py/r10694 Pywikipediabot/1.0 - -
 application/json
Crawler/1.2 - -
 text/..
cwikt-interwiki.py/r10768 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2665 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
redirect/r2612 Pywikipediabot/2.0 - -
 application/json
compat-pol.py/r10334 Pywikipediabot/1.0 - -
 application/json
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
www.monit24.pl-m24Bot/4.1- - -
 image/..
 text/..
redirect/r2602 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzredirectyeh.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2608 Pywikipediabot/2.0 - -
 application/json
compat-feedWikidata.py/r10655 Pywikipediabot/1.0 - -
 application/json
ko_missing2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
ContextAd Bot 1.0 - -
 text/..
 image/..
 application/ogg
SMNutchSpider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - -
 application/json
XLinkBot/1.00 - -
 text/..
pywikibot-compat-pwb.py/r10815 Pywikipediabot/1.0 - -
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
redirect/r2593 Pywikipediabot/2.0 - -
 application/json
pywikipedia-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Peachy MediaWiki Bot API Version 1.0 -
 application/vnd.php.serialized
strucr.com crawler 2.21.356 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 -
 application/json
pywikibot-1newar25.py/r10816 Pywikipediabot/1.0 - -
 application/json
 application/xml
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
HTMLParser/2.0 - -
 text/..
 image/..
SurakWare MediaWiki Bot/1.0 - -
 text/..
compat-touch.py/r10809 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/r2638 Pywikipediabot/2.0 - -
 application/json
redirect/r2659 Pywikipediabot/2.0 - -
 application/json
wAPI/1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
redirect/r2660 Pywikipediabot/2.0 - -
 application/json
redirect/r2594 Pywikipediabot/2.0 - -
 application/json
redirect/r2588 Pywikipediabot/2.0 - -
 application/json
bot-newar25.py/r10758 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zztakhasosi.py/r10694 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-aaa_elenco.py/r10761 Pywikipediabot/1.0 - -
 application/json
bot-pluralof.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
 application/json
strucr.com crawler 2.21.355 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
pyreza-mianwi.py/r11775 Pywikipediabot/1.0 -
 application/json
 application/xml
compat-feedWikidata-en.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
wikidata-sitelinks.py/r11618 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/r10815 Pywikipediabot/1.0 - -
 application/json
redirect/r2674 Pywikipediabot/2.0 - -
 application/json
maj_articles_recents/r2523 Pywikipediabot/2.0 - -
 application/json
logpatrol/r2522 Pywikipediabot/2.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
pywikibot-1newar25.py/r10802 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en,* -
 image/..
 text/..
porzucone/r2605 Pywikipediabot/2.0 - -
 application/json
pywikipedia-HBot_image_checker.py/r11775 Pywikipediabot/1.0 - -
 application/json
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-feedWikidataWithRDFFile.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
strucr.com crawler 2.21.358 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
bot arbre des mots ( mail address ) libcurl codeblock - -
 application/json
 text/..
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
bot arbre des mots ( mail address ) libcurl7.18.0 MSVC - -
 text/..
 application/json
redirect/r2604 Pywikipediabot/2.0 - -
 application/json
Python27-pythonw.exe/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
pywikibot-compat-aaa_categorieAutori.py/r10761 Pywikipediabot/1.0 - -
 application/json
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
 application/json
AdMedia bot - -
 text/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
redirect/r2568 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
 application/xml
 application/opensearchdescription+xml
mijnbots-OnvolDP.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-linktranslator.py/r10811 Pywikipediabot/1.0 - -
 application/json
redirect/r2556 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10809 Pywikipediabot/1.0 - -
 application/json
aleph-avbot.py/r11368 Pywikipediabot/1.0 -
 application/json
 text/..
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
run_replace.6r_2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
pywikipedia-fr.wikt.RemiseEnForme.py/r68 Pywikipediabot/1.0 - -
 application/json
redirect/r2617 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-importwdwiswv.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzhayati.py/r10694 Pywikipediabot/1.0 - -
 application/json
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - -
 application/xml
 text/..
cc_to_sitelinks/r2612 Pywikipediabot/2.0 - -
 application/json
porzucone/r2626 Pywikipediabot/2.0 - -
 application/json
strucr.com crawler 2.21.359 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
pwb/r2506 Pywikipediabot/2.0 - -
 application/json
authority2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-feedWikidata-fr.py/r10655 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
pywikibot-1newar25.py/r10786 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
redirect/r2577 Pywikipediabot/2.0 - -
 application/json
authority-de/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
2267809/r2638 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
bot-gender.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
commons_galleries_same_name/r2506 Pywikipediabot/2.0 - -
 application/json
D4SearchCrawler/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Full_update/r2554 Pywikipediabot/2.0 - -
 application/json
cwikt-interwiki.py/r10809 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-1newar25.py/r10819 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2657 Pywikipediabot/2.0 - -
 application/json
pywikibot-1newar25.py/r10768 Pywikipediabot/1.0 - -
 application/json
 application/xml
HRoestBot, de-wikipedia using pywikipedia framework -
 text/..
 application/json
wymowa/r2626 Pywikipediabot/2.0 - -
 application/json
Screaming Frog SEO Spider/2.22 - -
 text/..
 image/..
 audio/webm
pywikipedia-zzredirectyeh.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/r2638 Pywikipediabot/2.0 - -
 application/json
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
porzucone/r2669 Pywikipediabot/2.0 - -
 application/json
redirect/r2655 Pywikipediabot/2.0 - -
 application/json
2124312/r2605 Pywikipediabot/2.0 - -
 application/json
wymowa/r2605 Pywikipediabot/2.0 - -
 application/json
redirect/r2691 Pywikipediabot/2.0 - -
 application/json
pywikibot-_move2wikidata_taxon.py/r10617 Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/r2631 Pywikipediabot/2.0 - -
 application/json
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
all_pages/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
bot-null.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.57; ) - -
 text/..
 application/xml
pywikipedia-commonscat.py/r10694 Pywikipediabot/1.0 - -
 application/json
porzucone/r2616 Pywikipediabot/2.0 - -
 application/json
wymowa/r2638 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10797 Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/r2620 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10758 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-importwds.py/r10792 Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-newar25.py/r10766 Pywikipediabot/1.0 - -
 application/json
redirect/r2670 Pywikipediabot/2.0 - -
 application/json
php wikibot classes -
 application/vnd.php.serialized
 text/..
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
redirect/r2624 Pywikipediabot/2.0 - -
 application/json
Empedia Bot - -
 text/..
HersfoldIRCBot version 1.9.1 - -
 text/..
pywiki-zzporb-main.py/r10754 Pywikipediabot/1.0 - -
 application/json
Inlibris.com XMLBot/1.0 - -
 text/..
commons_categories_same_name/r2506 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 image/..
pywikipedia-git-wdph661sv.py/r90 Pywikipediabot/1.0 - -
 application/json
Roka Spider/Nutch-2.2.1 (' mail address ') - -
 text/..
compat-feedWikidata-nl.py/r10655 Pywikipediabot/1.0 - -
 application/json
pwb-klbot_wikdiata_name.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
refreshfeeds/r2638 Pywikipediabot/2.0 - -
 application/json
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/r10825 Pywikipediabot/1.0 - -
 application/json
delete/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
compat-featured.py/r10786 Pywikipediabot/1.0 - -
 application/json
check_db/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
bot-newar.py/r10716 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/r10782 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10809 Pywikipediabot/1.0 - -
 application/json
 application/xml
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
37.58.85.180 GoogleBot-Image/1.0 - -
 text/..
refreshfeeds/r2669 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.58; ) - -
 text/..
 application/xml
Yeobot/1.0 Contact: mail address ) - -
 text/..
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph661fr.py/r90 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
compat-featured.py/r10766 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-1newar25.py/r10806 Pywikipediabot/1.0 - -
 application/json
bot-newar.py/r10752 Pywikipediabot/1.0 - -
 application/json
logpatrol/r2623 Pywikipediabot/2.0 - -
 application/json
redirect/r2676 Pywikipediabot/2.0 - -
 application/json
compat-featured.py/r10794 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/r10761 Pywikipediabot/1.0 - -
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
gsa-crawler (Enterprise; T3-GMDVWEKT9WSQZ; mail address ) - -
 text/..
 image/..
PureBasic WikiBotFramework - d:User:InkoBot operated by d:User:Inkowik - -
 text/..
AnomieBOT 1.0 (IFDCloser; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-compat-pwb.py/r10805 Pywikipediabot/1.0 - -
 application/json
compat-basicarticledisamblist.py/r10754 Pywikipediabot/1.0 - -
 application/json
compat-importwdsit.py/r10792 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10815 Pywikipediabot/1.0 - -
 application/json
 application/xml
run_replace.6r_3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
QrobotW - -
 text/..
 image/..
logpatrol/r2659 Pywikipediabot/2.0 - -
 application/json
wymowa/r2669 Pywikipediabot/2.0 - -
 application/json
efcrawler/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 text/..
 image/..
pywikirish-welcome.py/r11215 Pywikipediabot/1.0 -
 application/json
compat-featured.py/r10752 Pywikipediabot/1.0 - -
 application/json
 application/xml
YisouSpider - -
 text/..
 application/json
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 -
 application/json
pwb/r2626 Pywikipediabot/2.0 - -
 application/json
compat-basicarticledisamblist.py/r10803 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
compat-featured.py/r10806 Pywikipediabot/1.0 - -
 application/json
redirect/r2627 Pywikipediabot/2.0 - -
 application/json
YBot/0.1 - -
 application/vnd.php.serialized
compat-touch.py/r10805 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/r2616 Pywikipediabot/2.0 - -
 application/json
pwb/r2659 Pywikipediabot/2.0 - -
 application/json
redirect/r2570 Pywikipediabot/2.0 - -
 application/json
pywikipedia3-welcome.py/r11747 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - -
 application/json
strucr.com crawler 2.21.357 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
language/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
verifcom/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-feedWikidata-en.py/r10655 Pywikipediabot/1.0 - -
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
pywikibot-compat-pwb.py/r10811 Pywikipediabot/1.0 - -
 application/json
User-Agent Baiduspider - -
 text/..
 image/..
dmashine-CleanupTematic.py/r10391 Pywikipediabot/1.0 - -
 application/json
descLabelCheck/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
sootv_param/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
compat-featured.py/r10417 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/r10815 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2663 Pywikipediabot/2.0 - -
 application/json
redirect/r2632 Pywikipediabot/2.0 - -
 application/json
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 image/..
 text/..
pwb/r2510 Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (CHUUClerk; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Spinuf Spider - -
 text/..
TptBot/1.0 by User:Tpt WikibasePhpLib/0.1 - -
 application/json
User:Bilderbot (derivativeFX) by Luxo on the Toolserver / PHP -
 application/vnd.php.serialized
 text/..
erfgoedbot-categorize_images.py/r11633 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-pwb.py/r10751 Pywikipediabot/1.0 - -
 application/json
 application/xml
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 text/..
cd-delinker.py/r11775 Pywikipediabot/1.0 -
 application/json
DotNetWikiBot/3.0 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
compat-featured.py/r10819 Pywikipediabot/1.0 - -
 application/json
 application/xml
cc_to_sitelinks/r2506 Pywikipediabot/2.0 - -
 application/json
porzucone/r2674 Pywikipediabot/2.0 - -
 application/json
pywikipedia-test.py/r11775 Pywikipediabot/1.0 - -
 application/json
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
porzucone/r2655 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10768 Pywikipediabot/1.0 - -
 application/json
 application/xml
addDescGemeinde/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
porzucone/r2604 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
big_data_pull_by_subject/r2160 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10805 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-${maven.build.timestamp} https://archive.org/details/archive.org_bot) - -
 text/..
 image/..
 application/xml
redirect/r2614 Pywikipediabot/2.0 - -
 application/json
porzucone/r2684 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - -
 text/..
pywikibot-compat-commonscat.py/r10810 Pywikipediabot/1.0 - -
 application/json
pwb/r2670 Pywikipediabot/2.0 - -
 application/json
unique-merge/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
redirect/r2590 Pywikipediabot/2.0 - -
 application/json
niki-bot - -
 text/..
 image/..
pwb/r2691 Pywikipediabot/2.0 - -
 application/json
porzucone/r2594 Pywikipediabot/2.0 - -
 application/json
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
pywikipedia-zzaddnavebox.py/r10694 Pywikipediabot/1.0 - -
 application/json
porzucone/r2577 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzfawikipatroler.py/r10694 Pywikipediabot/1.0 - -
 application/json
CoBot/1.0 (Olivaw) - -
 application/json
python-wikitools/1.2 (User:LaraBot) -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
pywikibot-compat-commonscat.py/r10766 Pywikipediabot/1.0 - -
 application/json
refresh_merk/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
porzucone/r2602 Pywikipediabot/2.0 - -
 application/json
porzucone/r2593 Pywikipediabot/2.0 - -
 application/json
porzucone/r2652 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10816 Pywikipediabot/1.0 - -
 application/json
porzucone/r2612 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10806 Pywikipediabot/1.0 - -
 application/json
archiv-erl-bot/r2625 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10778 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 text/..
 -
hivaBot/hivaBot-0.2 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
pywikibot-compat-commonscat.py/r10761 Pywikipediabot/1.0 - -
 application/json
compat-feedWikidata-de.py/r10655 Pywikipediabot/1.0 - -
 application/json
ybot-compat-cat_copy.py/r10718 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-compat-commonscat.py/r10753 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-touch.py/r10792 Pywikipediabot/1.0 - -
 application/json
 application/xml
commons-imagerecat.py/r11028 Pywikipediabot/1.0 - -
 application/json
Zing-BottaBot/1.0 - -
 text/..
pywikibot-compat-commonscat.py/r10769 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-featured.py/r10742 Pywikipediabot/1.0 - -
 application/json
pwb/r2612 Pywikipediabot/2.0 - -
 application/json
porzucone/r2660 Pywikipediabot/2.0 - -
 application/json
MediaWiki::Bot/5.005007 - -
 application/json
redirect/r2680 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10792 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/r10819 Pywikipediabot/1.0 - -
 application/json
compat-feedWikidata-it.py/r10655 Pywikipediabot/1.0 - -
 application/json
pywikipedia-amb_mehrdadbot.py/r11775 Pywikipediabot/1.0 - -
 application/json
porzucone/r2608 Pywikipediabot/2.0 - -
 application/json
porzucone/r2665 Pywikipediabot/2.0 - -
 application/json
pywikipedia-pwb.py/r10805 Pywikipediabot/1.0 - -
 application/json
 application/xml
 image/..
 text/..
pywikibot-compat-commonscat.py/r10765 Pywikipediabot/1.0 - -
 application/json
porzucone/r2556 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
pwb/r2684 Pywikipediabot/2.0 - -
 application/json
bot-formof.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/r10811 Pywikipediabot/1.0 - -
 application/json
 application/xml
CRAWLERANALYTICS - -
 image/..
 text/..
porzucone/r2623 Pywikipediabot/2.0 - -
 application/json
redirect/r2607 Pywikipediabot/2.0 - -
 application/json
Erel Bot - -
 text/..
('python-wikitools/1.1.1 (User:BernsteinBot)',) - -
 application/json
porzucone/r2588 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
pywiki-zzporb-main.py/r10811 Pywikipediabot/1.0 - -
 application/json
pywiki-newar25.py/r10811 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/r10754 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/r10786 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (PUICloser; see [[User:AnomieBOT]]) - -
 application/json
pwb/r2528 Pywikipediabot/2.0 - -
 application/json
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-pwb.py/r10751 Pywikipediabot/1.0 - -
 application/json
 application/xml
 image/..
 text/..
refreshfeeds/r2652 Pywikipediabot/2.0 - -
 application/json
pywikipedia-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
compat-feedWikidata-fr.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
Pywikipedia-anniversary.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
wymowa/r2620 Pywikipediabot/2.0 - -
 application/json
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
compat-basicarticledisamblist.py/r10758 Pywikipediabot/1.0 - -
 application/json
cwikt-interwiki.py/r10708 Pywikipediabot/1.0 - -
 application/json
 application/xml
gsa-crawler (Enterprise; T4-KW9PXNCULXJSR; mail address ) - -
 text/..
compat-basicarticledisamblist.py/r10806 Pywikipediabot/1.0 - -
 application/json
pywikipedia-com_wrong_info.py/r10458 Pywikipediabot/1.0 - -
 application/json
 text/..
porzucone/r2568 Pywikipediabot/2.0 - -
 application/json
D-Spider / 1.0 -
 text/..
pywikipedia-git-new_iw.py/r90 Pywikipediabot/1.0 - -
 application/json
GoogleBot - -
 text/..
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
wymowa/r2610 Pywikipediabot/2.0 - -
 application/json
pwb-klbot_wikdiata_name22.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
redirect/r2592 Pywikipediabot/2.0 - -
 application/json
wymowa/r2593 Pywikipediabot/2.0 - -
 application/json
pywikipedia-touchf.py/r10458 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2591 Pywikipediabot/2.0 - -
 application/json
update-regular-users/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10825 Pywikipediabot/1.0 - -
 application/json
compat-redirect.py/r10417 Pywikipediabot/1.0 - -
 application/json
wymowa/r2659 Pywikipediabot/2.0 - -
 application/json
CheMoBot/1.00 - -
 text/..
ko_missing/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
My Nutch Spider/Nutch-2.2.1 - -
 text/..
wymowa/r2618 Pywikipediabot/2.0 - -
 application/json
HTMLParser/1.6 - -
 text/..
29101.69total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Feb 10, 2014 14:08
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers