Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Mar 2014 - 31 Mar 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 123,192,030 page requests (mime type text/html only!) per day are considered crawler requests, out of 625,405,580 external requests, which is 19.7%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-(compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)Accept: text/html,application/xhtmlxml,application/xml;image/png,image/jpeg,image/*;q=0.9,*/*;q=0.8 en-us;q=0.7, en;q=0.3 -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~doingfly) - -
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/p/ldspider/wiki/Robotstext/..ldspider (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2; rv:22.0) Gecko/20100101 Firefox/22.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~phproxypro) zh-CN,zh;q=0.8,en;q=0.6 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/p/ldspider/wiki/Robotsapplication/rdf+xmlldspider (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~certificate-check-403) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/p/ldspider/wiki/Robots-ldspider (url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: s~certificate-check-403) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.117 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066a) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066a) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~imgdaili3) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066a) zh-CN,zh;q=0.8 -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..Wiki.java 0.30 AppEngine-Google; (url; appid: s~wikipediatools-hrd) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ucsm111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.117 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066a) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gregatr0n) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appengine-Wiki.java 0.30 AppEngine-Google; (url; appid: s~wikipediatools-hrd) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icebre4ker) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 KHTML Chrome/33.0.1750.117 Safari/537.36 AppEngine-Google; (url; appid: s~nsftcn) zh-CN,zh;q=0.8 -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0 AppEngine-Google; (url; appid: s~ad0066f) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: scraperproxy) - -
 code.google.com/appenginetext/..Acre/dev/59:906 ubiquity.freebaseapps.com AppEngine-Google; (url; appid: s~freebaseapps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nmimsforti) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vemmastar-c4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ahsanovic) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~projection11111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vadim-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nuraiitj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go6internet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: student-pupil-help) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn03) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ikaryse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: madkaiser2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~e-ddos) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-data) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kushgenius) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 www.google.com/bot.htmlapplication/oggMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: red-arg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~palenquegprs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jttm-server-prox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 KHTML Chrome/33.0.1750.117 Safari/537.36 AppEngine-Google; (url; appid: s~nsftcn) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitalhunch-freenet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~thephotoss-hrd) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hoptheborder) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 KHTML Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (url; appid: s~feedly-social) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zmomyanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icecool1987) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yblk-ext-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~policevan2) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-(compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US -
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.html-(compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmlimage/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmlimage/..ozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) SL -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
msn
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - https=1
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)1 readers - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/robot/-(compatible; AhrefsBot/5.0; url) -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12-1-amd64; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.12-1-amd64; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..zerbybot (freeworld-global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-45-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; GMT04:00/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.6-1-ARCH; java 1.7.0_51; Europe/nb) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.3.1.el6.x86_64; java 1.6.0_30; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-12-generic; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-18-generic; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-17-generic; java 1.7.0_51; Asia/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-18-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; GMT04:00/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12-1-amd64; java 1.7.0_21; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.11.0-18-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.html-zerbybot (freeworld-global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.13.6-1-ARCH; java 1.7.0_51; Europe/nb) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.5.1.el6.x86_64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.6-gentoo; java 1.7.0_51; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.13.5-gentoo; java 1.7.0_45; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.6-gentoo; java 1.7.0_45; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.5-gentoo; java 1.7.0_45; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.13.6-gentoo; java 1.7.0_45; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.13-1-686-pae; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; i386 Linux 3.13-1-686-pae; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-46-generic; java 1.6.0_30; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.7.10-1.28-default; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.5.1.el6.x86_64; java 1.6.0_30; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Australia/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.11.0-18-generic; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmlimage/..yacybot (freeworld/global; amd64 Linux 3.12-1-amd64; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_51; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-59-generic; java 1.7.0_51; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.10.25-gentoo; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.13.6-gentoo; java 1.7.0_51; Europe/pl) url en-us,en;q=0.5 -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 https=1
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
commoncrawl
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 commoncrawl.org/faq/-CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 commoncrawl.org/faq/text/..CCBot/2.0 (url) - -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
 nodejs.orgapplication/jsonNodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
aramabeta
 www.aramabeta.comtext/..betasearch/Nutch-2.2.1 (Academic Beta Search; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.aramabeta.comtext/..betasearch/Nutch-2.2.1 (Academic Beta Search; url; mail address ) - -
 www.aramabeta.com-betasearch/Nutch-2.2.1 (Academic Beta Search; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.aramabeta.comtext/..betasearch/Nutch-2.2.1 (Academic Beta Search; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 https=1
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation-WPCleaner (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.22.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggle-Huggle/2.1.22.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.22.0 url -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
sblog
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) - -
 fulltext.sblog.cz/-Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.19 - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
wordpress
 y35pm.wordpress.comtext/..WordPress/3.8.1; url - -
 thesandymonocle.wordpress.comtext/..WordPress/3.8.1; url - -
 nosce.wordpress.comtext/..WordPress/3.8.1; url - -
 whatownsme.wordpress.comtext/..WordPress/3.8.1; url - -
 menschlein.wordpress.comtext/..WordPress/3.8.1; url - -
 opensourceenterprise.wordpress.comtext/..WordPress/3.9-beta2; url - -
 hoangkimlong.wordpress.comtext/..WordPress/3.8.1; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.8.1; url - -
 manofroma.wordpress.comtext/..WordPress/3.8.1; url - -
 homepaddock.wordpress.comtext/..WordPress/3.8.1; url - -
 urielarte.wordpress.comtext/..WordPress/3.8.1; url - -
 log4aman.wordpress.comtext/..WordPress/3.8.1; url - -
 drstvmovies.wordpress.comtext/..WordPress/3.8.1; url - -
 nosce.wordpress.comtext/..WordPress/3.9-beta2-27714; url - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url -
bin-co
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/-Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; newscoccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-132-ga762834 (url) en -
 scrapy.orgtext/..Scrapy/0.22.2 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-77-gc99f1d2 (url) en -
 scrapy.org-Scrapy/0.23.0-132-ga762834 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-158-gf687455 (url) en -
 scrapy.org-Scrapy/0.23.0-77-gc99f1d2 (url) en -
 scrapy.org-Scrapy/0.22.2 (url) en -
 scrapy.org-Scrapy/0.23.0-158-gf687455 (url) en -
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
flipboard
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) ru -
 www.majestic12.co.uk/bot.php?text/..(compatible; MJ12bot/v1.4.4; url) - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * https=1
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ru -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics - url - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
pinterest
 pinterest.com/image/..Pinterest/0.1 url - -
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/-Pinterest/0.1 url - -
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
 archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..(compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
 www.diffbot.comtext/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
 www.phonifier.com-Mozilla/5.0 (compatible; Phonifier; url) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
github
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 -
 github.com/adrienkohlbecker/projetosyapplication/jsonProjetOSY/1.0 (url; mail address ) - -
 github.com/adrienkohlbecker/projetosy-ProjetOSY/1.0 (url; mail address ) - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/edsu/linkypediaapplication/jsonlinkpyediabot v0.1: url - -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/2.1.1p76 (url) en-us,en;q=0.5 -
 github.com/pauldix/feedzirra/tree/master-feedzirra url - -
xbmc
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.2;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Linux; Android; 3.0.36 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130527-a36a92b (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (iOS; 13.0.0, Version 6.1.4 (Build 10B809); url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 5.1; url) - -
example
 example.comapplication/jsonArticle-Level Metrics - url - -
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolTool/text/..User-Agent: MyCoolTool/1.1 (url; mail address ) - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.com/MyCoolTool/application/jsonMyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
 example.com/MyCoolTool/text/..MyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 -
automoveisnaweb
 automoveisnaweb.com.br/uk5text/..WordPress/3.8.1; url - -
 automoveisnaweb.com.br/uk4text/..WordPress/3.8; url - -
 automoveisnaweb.com.br/uk6text/..WordPress/3.8.1; url - -
 automoveisnaweb.com.br/uk3text/..WordPress/3.8.1; url - -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.comtext/..FeedlyApp/1.0 (url) - -
localhost
 localhost/clumpit/bot.phptext/..Mozilla/5.0 (compatible; clumboot; url) - -
 localhost/index/bot.phptext/..Mozilla/5.0 (compatible; index_robot; url) - -
 localhost/wordpresstext/..WordPress/3.8.1; url - -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; ChinasoSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) -
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) -
 toolserver.org/~dispenser/text/..DispensersTools (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
domeinwinkel
 www.domeinwinkel.nl/text/..Domeinwinkel CatchCrawler/1.0 (url) - -
wesee
 www.wesee.com/en/support/bot/image/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/en/support/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
temposemetodos
 temposemetodos.com.br/up1text/..WordPress/3.8; url - -
 temposemetodos.com.br/up4text/..WordPress/3.8; url - -
 temposemetodos.com.br/up2text/..WordPress/3.8; url - -
 temposemetodos.com.br/up6text/..WordPress/3.8; url - -
 temposemetodos.com.br/up3text/..WordPress/3.8; url - -
 temposemetodos.com.br/up5text/..WordPress/3.8; url - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
crossref
 alm.labs.crossref.orgapplication/jsonArticle Level Metrics - url -
site
 www.site.com/application/jsonBot/2.0 (url) -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
http:
 http://www.goguide-directory.com.au/application/vnd.php.serializedurl - -
 http://alm.example.orgapplication/jsonArticle Level Metrics - url - -
 http://www.goguide-directory.com.au/application/vnd.php.serializedTesting for url - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
 tweetmeme.com/image/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
trendiction
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
wikidict
 www.wikidict.detext/..url - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/text/..Drupal (url) -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
owasp
 www.owasp.org/index.php/Category:OWASP_DirBuster_Projecttext/..DirBuster-1.0-RC1 (url) - -
import
 import.iotext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
 www.rcdtokyo.com/pc2m/-Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url) -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) ja -
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de -
jyvee
 www.jyvee.comapplication/xmlFeedFetcher; url; v.1.3.79; - -
 www.jyvee.comtext/..FeedFetcher; url; v.1.3.79; - -
mapsway
 mapsway.com/text/..MapswayBot/1.0 (url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
 support.paper.li/entries/20023257-what-is-paper-li-Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
Anonymouse
 Anonymouse.org/text/..url (Unix) - -
 Anonymouse.org/image/..url (Unix) - -
mytest
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131104-0244 url) - -
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131104-0244 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
qbqm
 www.qbqm.fr/text/..QBQM/1.0 (url) fr -
allbookstores
 www.allbookstores.comapplication/jsonurl - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
gov
 pandora.nla.gov.au/crawl.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot/3.3.0 url) - -
 pandora.nla.gov.au/crawl.html-Mozilla/5.0 (compatible; archive.org_bot/3.3.0 url) - -
 pandora.nla.gov.au/crawl.htmltext/..Mozilla/5.0 (compatible; archive.org_bot/3.3.0 url) - -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.0.9; url ) en-us,en;q=0.5 -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.0.7; url ) - -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.0.9; url ) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
superfeedr
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.com-Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
 www.okian.ro/application/jsonMyBot/1.0 (url) - -
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
bit
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) de
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) en
nvdev
 qa.nvdev.comtext/..Netvibes (url) - -
oblivion
 www.oblivion.im/search/y/bot.phptext/..Mozilla/5.0 (compatible; Please Name Your robot; url) - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
softwaretipsandadvice
 softwaretipsandadvice.comtext/..WordPress/3.8.1; url - -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
linkdex
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
netnewswireapp
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
 netnewswireapp.com/mac/image/..NetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) fr-fr -
profound
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - -
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
botify
 botify.comtext/..Mozilla/5.0 (compatible; botify; url) - -
swish-e
 swish-e.org/text/..swish-e url - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
tineye
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
willmartin
 willmartin.comtext/..WordPress/3.8.1; url - -
 willmartin.com-WordPress/3.8.1; url - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.11 (url) - -
watchmouse
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
tiscali
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.18.81 url) - -
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.10.2 url) * -
backgroundswitcher
 www.backgroundswitcher.com/text/..John's Background Switcher 4.8 (url) - -
 www.backgroundswitcher.com/image/..John's Background Switcher 4.8 (url) - -
plagsniffer
 plagsniffer.de/application/jsonPlagsniffer (url) -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
servicesonrequest
 www.servicesonrequest.comtext/..User-Agent: WikiBot (url) - -
 www.servicesonrequest.com/application/jsonWikiBot(url) - -
weserv
 images.weserv.nl/image/..Mozilla/5.0 (compatible; ImageFetcher/5.5; url) -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot; url) - -
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot; url) -
simplepie
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
axtaris
 axtaris.orgtext/..Axtaris Web Crawler/Axtaris-1.0.1 (Hello from Axtaris.org | Web-Crawler; url; mail address ) - -
QtWeb
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.3 (url) -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
com?type=714647&fr=spigot-yhp-ch
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
scarecrow
 www.scarecrow.com/application/jsonScarecrow/1.0 (url; mail address ) - -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo-crawler; url; mail address ) - -
 www.parsijoo.ir/image/..Mozilla/5.0 (compatible; parsijoo-crawler; url; mail address ) - -
besaba
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
datoid
 datoid.cz/tag-information/application/jsonDatoid/0.9 (url; mail address ) BasedOnNette/2.0.9 - -
image2play
 image2play.com/image/..Mozilla/5.0 (compatible; Image2play/0.1; url) - -
blogspot
 beh1001.blogspot.com/p/the-bot.htmltext/..Mozilla/5.0 (compatible; Beh1001/beta; url) - -
 alberapps.blogspot.com-TiempoBus/2.8 (url; mail address ) - -
 alberapps.blogspot.comtext/..TiempoBus/2.8 (url; mail address ) - -
creativepulses
 creativepulses.nltext/..CreativePulses Crawler (url) - -
vientech
 vientech.at/abuse.txttext/..url - -
thearchangelmichael
 thearchangelmichael.nettext/..WordPress/3.4.2; url - -
cinemahelp
 cinemahelp.us/text/..CinemahelpBot/1.0 (url) -
languagetool
 community.languagetool.orgapplication/xmlurl bot, contact: mail address - -
 community.languagetool.orgtext/..url bot, contact: mail address - -
1:8080
 127.0.0.1:8080/au2text/..WordPress/3.8.1; url - -
 127.0.0.1:8080/tileshoptext/..WordPress/3.8.1; url - -
177307.339999975total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 7) - -
 application/vnd.php.serialized
 text/..
 -
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
pwb/rg2746 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
tigerbot - -
 -
 application/json
 text/..
 image/..
maj_articles_recents/rg2805 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
LinkParser/2.0 - -
 text/..
pywikipedia-rad1.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
 -
pywikipedia-rad1.py/r10694 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
ClueBot/2.0 - -
 application/vnd.php.serialized
maj_articles_recents/rg2754 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
wikiwix-bot-3.0 - -
 text/..
redirect/rg2781 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/rg2800 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/json
 -
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 -
 text/..
 application/x-www-form-urlencoded
internationalization/r2631 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
pywikipedia-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
 application/x-www-form-urlencoded
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN -
 text/..
 -
 image/..
pywikipedia2-addzumra.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
pywikibot-1newar25.py/rg10901 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
pywikipedia-git-wdph94.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
PythonWikipediaBot/1.0 - -
 application/json
 text/..
 -
 application/x-www-form-urlencoded
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 text/..
 -
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 -
 application/ogg
redirect/rg2805 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/rg2803 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
harvest_nltaxo/rg2726 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
MediaWiki::Bot/5.005006 - -
 application/json
pywikibot-1newar25.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
MediaWiki::Bot/3.2.6 - -
 application/json
pywikipedia-radeh.py/r10694 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
run_replace.7r/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
run_pagefromfile.4/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Twitterbot/1.0 - -
 text/..
 image/..
 -
 application/pdf
 application/ogg
 audio/webm
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
PythonWikipediaBot/1.0 -
 application/json
 -
 text/..
wiktionary-z_paginasrequeridas.py/r10529 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
redirect/rg2815 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzfatranslator.py/r10694 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 -
 application/json
MediaWiki::Bot/3.005002 - -
 application/json
redirect/rg2816 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
ClueBot/1.1 - -
 application/vnd.php.serialized
DigitalsmithsBot - -
 text/..
harvest_nltaxo/r2458 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
run_pagefromfile.6/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
 application/json
BOT/0.1 (BOT for JCE) - -
 text/..
 application/x-www-form-urlencoded
run_pagefromfile.5/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
 application/x-www-form-urlencoded
pwb/rs3943 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
 application/x-www-form-urlencoded
HTMLParser/2.0 - -
 application/x-wiki
 text/..
 -
Semantix Bot 0.1 - -
 text/..
pwb/rg2823 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
spider - -
 text/..
 application/json
 application/vnd.php.serialized
 image/..
 -
compat-checkimages.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
redirect/rg2777 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
pywikibot-1newar25.py/r10873 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
pywikipedia2-zzfatranslator.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
360spider-image - -
 image/..
 text/..
ContextAd Bot 1.0 - -
 text/..
 image/..
compat-feedWikidata-en.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-featured.py/r10441 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
 application/x-www-form-urlencoded
run_replace.7r.3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
batcher/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
addDescMensch/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
Webbot - -
 text/..
 -
WikidataBot framework - -
 application/json
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rg2807 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/r10854 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
SAS Web Crawler - -
 text/..
pwb/r2772 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
run_replace.7r.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/r2669 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
 text/..
redirect/rg2772 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 application/json
pwb/rs3850 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
wikiparser/1 CFNetwork/673.2.1 Darwin/13.1.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
DotNetWikiBot/3.0 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 application/x-www-form-urlencoded
 text/..
pywikipedia-zzinter.py/r10694 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
pywiki-cfd.py/r10360 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikibot-1newar25.py/r10870 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
pywikibot-1newar25.py/r10860 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikibot-compat-commonscat.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pwb/rs3828 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-addzumra.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywiki-newar25.py/rg10901 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
pywiki-newar25.py/rg10893 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywikibot-1newar25.py/rg10878 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
porzucone/rg2915 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rs3844 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
redirect/rg2788 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
createHierarchy/rg2781 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 text/..
 -
 application/x-www-form-urlencoded
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
redirect/rg2776 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-newar.py/r10523 Pywikipediabot/1.0 - -
 application/json
 application/xml
SentiBot www.sentibot.eu (compatible with GoogleBot) - -
 text/..
 application/xml
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
pyreza-zz_make_cat.py/r10778 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-git-maghalesaz2.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
mail address mail address \xE2\x80\x93 MediaWiki Tcl Bot Framework 0.5 - -
 application/x-www-form-urlencoded
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
createHierarchy/rg2825 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
 text/..
GermCrawler - -
 application/json
 application/xml
 text/..
pyreza-zz_make_cat.py/r10778 Pywikipediabot/1.0 - -
 application/json
 -
election/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
JackBot-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywiki-linktranslator.py/r10870 Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia2-checkimages.py/r10550 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 text/..
www.integromedb.org/Crawler - -
 text/..
 application/xml
 application/json
pwb/rs3931 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-newar.py/r10523 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.54; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
redirect/rg2798 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
redirect/rg2778 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
 -
pywikibot-compat-commonscat.py/r10870 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
porzucone/rg2781 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Yeobot/1.1 Contact: mail address ) - -
 text/..
pwb/rs3833 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikibot-compat-commonscat.py/rg10901 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
WorldOfMusicBetaBot/1.0 - -
 text/..
porzucone/rg2800 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
 video/webm
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 - -
 application/json
 application/xml
 -
GoogleBot-Image/1.0 -
 text/..
 image/..
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - -
 application/json
 text/..
 -
pwb/rg2784 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
milog_bot/1.0 ( mail address ) - -
 text/..
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
 -
ko_missing/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Healing For Me Robot - -
 application/json
pwb/rs3939 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
wymowa/rg2915 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
My Nutch Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 application/ogg
pywikipedia2-addgalleryurtest.py/r10866 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
compat-redirect.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-zz_make_cat.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
DotNetWikiBot/2.100 (Unix 3.2.0.56; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
Mozilla/5.0 (compatible; SputnikBot/2.3) ru,en;q=0.1,*;q=0.01 -
 text/..
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
KatBot bot.nytlabs.com> - -
 text/..
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
Mozilla/5.0 (compatible; FlappyBot; ) - -
 text/..
 image/..
pywikibot-compat-commonscat.py/rg10880 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
pywikipedia-zzporb-main.py/r10694 Pywikipediabot/1.0 - -
 application/json
pwb/rs3804 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikibot-compat-commonscat.py/r10873 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
pywikipedia-zzporb-main.py/r10694 Pywikipediabot/1.0 -
 -
wymowa/rg2781 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
logpatrol/rg2781 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 -
 application/json
compat-basictemplatedisamblist.py/rg10895 Pywikipediabot/1.0 -
 -
cwikt-interwiki.py/r10870 Pywikipediabot/1.0 - -
 application/json
 application/xml
ProoXiBot fr -
 text/..
 -
wymowa/rg2800 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikibot-1newar25.py/rg10880 Pywikipediabot/1.0 - -
 application/json
 -
cwikt-interwiki.py/r10870 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
HTMLParser/1.6 - -
 text/..
 -
 image/..
pywikipedia-git-maghalesaz.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-basictemplatedisamblist.py/rg10880 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-basictemplatedisamblist.py/r10866 Pywikipediabot/1.0 -
 -
compat-basictemplatedisamblist.py/rg10880 Pywikipediabot/1.0 -
 -
compat-basictemplatedisamblist.py/rg10895 Pywikipediabot/1.0 - -
 application/json
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-basictemplatedisamblist.py/r10866 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 -
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
run_replace.7r.4/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikibot-compat-aaa_elenco.py/r10761 Pywikipediabot/1.0 - -
 application/json
 -
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
 application/xml
TrueKnowledgeBot bot mail address > - -
 application/xml
 application/vnd.php.serialized
 image/..
COIBot/1.00 - -
 text/..
 -
redirect/rg2783 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
 image/..
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
 application/x-www-form-urlencoded
YisouSpider - -
 text/..
 application/xml
compat-checkimages.py/rs10878 Pywikipediabot/1.0 - -
 -
 application/json
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 en-us -
 image/..
 text/..
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rs3864 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikibot-compat-commonscat.py/rg10878 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - -
 application/json
aleph-avbot.py/r11368 Pywikipediabot/1.0 -
 application/json
 text/..
 -
2807691/rg2781 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
XLinkBot/1.00 - -
 text/..
 -
pywikipedia-git-zher.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
GoogleBot-Image/1.0 - https=1
 text/..
 -
pywikibot-compat-commonscat.py/rg10886 Pywikipediabot/1.0 - -
 application/json
 -
porzucone/rg2805 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.100 (Unix 3.2.0.58; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
pywikipedia-zzredirectyeh.py/r10694 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikibot-compat-commonscat.py/rg10899 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
67975/rg2863 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
pwb/rs3816 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
harvest_nltaxo/rg2781 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-zzredirectyeh.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-pwb.py/r10860 Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
 application/xml
 image/..
 text/..
pwb/rs3827 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
GoogleBot-Image/1.0 - https=1
 text/..
pywikipedia-zztakhasosi.py/r10694 Pywikipediabot/1.0 -
 -
ht/rs3833 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
SurakWare MediaWiki Bot/1.0 - -
 application/x-www-form-urlencoded
 text/..
Peachy MediaWiki Bot API Version 1.0 -
 application/vnd.php.serialized
chrwikt/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
CaBot - -
 text/..
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
 -
Wikibot/2.1.1 CFNetwork/672.1.13 Darwin/14.0.0 en-us -
 image/..
 text/..
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
TambonBot Wikibase.NET/0.1 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-fr.wikt.RemiseEnForme.py/r68 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikipedia-new_interwiki.py/r10866 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pwb/rs3916 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rs3841 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rg2772 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rs3922 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-aaa_categorieAutori.py/r10761 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
compat-featured.py/r10866 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-zztakhasosi.py/r10694 Pywikipediabot/1.0 - -
 application/json
wAPI/1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
pywiki-linktranslator.py/rg10893 Pywikipediabot/1.0 - -
 application/json
 -
pwb/rg2800 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
refreshfeeds/rg2781 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia2-welcome.py/r10550 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - -
 application/xml
 text/..
pwb/rs3919 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-sichteroptin.py/r10880 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 text/..
 application/json
 -
recent/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
refreshfeeds/rg2915 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Kolega2357-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
 -
cwikt-interwiki.py/r10854 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/rg2803 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-zzfatranslator.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
 application/x-www-form-urlencoded
compat-noreferences.py/r10873 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
compat-featured.py/r10866 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-basicarticledisamblist.py/rg10901 Pywikipediabot/1.0 - -
 application/json
cwikt-interwiki.py/r10854 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pyreza-zzcity_article.py/r10778 Pywikipediabot/1.0 - -
 application/json
 application/xml
slogi_simple/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider; zh-CN -
 text/..
 -
 image/..
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - -
 application/json
 -
ITeco bot, v0.0, mail address - -
 -
 text/..
pyreza-zzcity_article.py/r10778 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pwb/rs3809 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
strucr.com crawler 2.22.361 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
pywikipedia-git-maghalesaz2.py/r123 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
TripAdvisor-WikipediaGeoRequestBot/0.1 ( mail address ) - -
 application/json
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 image/..
 -
 application/ogg
pwb/rs3831 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/r2526 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 -
pwb/rs3800 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-basicarticledisamblist.py/rg10901 Pywikipediabot/1.0 -
 -
logpatrol/rg2816 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-commonscat.py/r10786 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
compat-noreferences.py/r10873 Pywikipediabot/1.0 - -
 application/json
 application/xml
refreshfeeds/rg2800 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-1newar25.py/rg10899 Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia-git-labelfixer.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
ci/rs3833 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
COIBot/2.0 - -
 text/..
py-replace.py/r10589 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
 -
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 -
 text/..
logpatrol/rg2905 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pyreza-check_p.py/r10778 Pywikipediabot/1.0 - -
 application/json
compat-basicarticledisamblist.py/r10866 Pywikipediabot/1.0 -
 -
pywikipedia-zzaddnavebox.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
compat-basicarticledisamblist.py/rg10880 Pywikipediabot/1.0 - -
 application/json
pywikipedia-cityarticles3.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-basicarticledisamblist.py/r10866 Pywikipediabot/1.0 - -
 application/json
mail address - -
 application/json
 text/..
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) -
 application/x-www-form-urlencoded
compat-basicarticledisamblist.py/rg10880 Pywikipediabot/1.0 -
 -
porzucone/rg2905 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
HRoestBot, de-wikipedia using pywikipedia framework -
 text/..
 application/json
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
pywikipedia-zzhayati.py/r10694 Pywikipediabot/1.0 - -
 application/json
Inlibris.com XMLBot/1.0 - -
 text/..
Kolega2357-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
run_replace.6r_2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
porzucone/rg2899 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
CorenSearchBot/1.7 en libwww-perl/6.03 - -
 text/..
 application/x-www-form-urlencoded
pywikipedia-zzhayati.py/r10694 Pywikipediabot/1.0 -
 -
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
porzucone/rg2772 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-touch.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 application/xml
 -
porzucone/rg2889 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pyreza-check_p.py/r10778 Pywikipediabot/1.0 -
 -
redirect/rg2807 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg2798 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg2816 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
redirect/rg2799 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-_move2wikidata_np.py/r10829 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
porzucone/rg2778 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
pywikibot-compat-commonscat.py/r10876 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
By IdeasBot Kernel, Version:0.9.0.10 - -
 text/..
compat-checkimages.py/rs10872 Pywikipediabot/1.0 - -
 application/json
 -
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - -
 text/..
 -
weave_rest_bot, contact mail address - -
 text/..
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - -
 application/xml
 text/..
 -
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en,* -
 image/..
 text/..
YBot/0.1 - -
 application/vnd.php.serialized
porzucone/rg2777 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-commonscat.py/r10860 Pywikipediabot/1.0 - -
 -
 application/json
bot-null.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
porzucone/rg2815 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
porzucone/rg2911 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-redirect.py/rs10880 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
porzucone/rg2803 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikibot-compat-commonscat.py/rg10895 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
pywikipedia-zzfawikipatroler.py/r10694 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
iw_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - -
 text/..
 application/json
 application/xml
pywikipedia-commonscat.py/r10694 Pywikipediabot/1.0 -
 -
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
pywikipedia-zzgallery.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-featured.py/rg10880 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
User-Agent Baiduspider - -
 text/..
 image/..
Full_update/r2554 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywiki-linktranslator.py/r10865 Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia-addgalleryurtest.py/r10860 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/rg10901 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-newar25.py/r10870 Pywikipediabot/1.0 - -
 -
 application/json
compat-featured.py/rg10901 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
By IdeasBot Framework, Version:1.0.460 - -
 text/..
pwb/rg2815 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-checkimages.py/rs10865 Pywikipediabot/1.0 - -
 application/json
 -
PHPCrawl - -
 text/..
 -
voverb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-featured.py/r10870 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-commonscat.py/r10694 Pywikipediabot/1.0 - -
 application/json
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
Pywikibot-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
 -
wymowa/rg2913 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikilab-absents.py/r11775 Pywikipediabot/1.0 -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
 -
pwb/rs3892 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
gobotbeta de,de-DE,en,en-US -
 text/..
pywikipedia-git-wdph665.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-featured.py/r10873 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/r10870 Pywikipediabot/1.0 - -
 application/json
add_text/rg2911 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-noreferences.py/r10876 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
addDescCat/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 -
 application/json
createHierarchy/rg2764 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
postWiki/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-git-wdph100.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
wymowa/rg2788 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
wymowa/rg2776 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rs3933 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-git-new_iw14.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Your Nutch Spider/Nutch-2.2.1 - -
 text/..
 application/opensearchdescription+xml
compat-featured.py/rg10899 Pywikipediabot/1.0 - -
 application/json
all_pages/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
wymowa/rg2816 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
dumpprocessor/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-replace.py/r10833 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
cd-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10873 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
User:Bilderbot (derivativeFX) by Luxo on the Toolserver / PHP -
 application/vnd.php.serialized
 text/..
pwb/rs3826 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Erel Bot - -
 text/..
compat-featured.py/rg10880 Pywikipediabot/1.0 - -
 application/json
bot-audiolang.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/3.10 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
compat-featured.py/rg10899 Pywikipediabot/1.0 -
 -
wymowa/rg2905 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-git-iwdeler.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
By IdeasBot Kernel, Version:0.9.0.9 - -
 text/..
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 text/..
 image/..
wymowa/rg2903 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-noreferences.py/r10876 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/rg10878 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/rg2805 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
 application/pdf
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
pywikipedia-vyzreplace.py/r11781 Pywikipediabot/1.0 - -
 -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 text/..
GoogleBot (gocrawl v0.4) - -
 text/..
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en-US,* -
 image/..
 text/..
cwikt-interwiki.py/rg10901 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-basictemplatedisamblist.py/rg10907 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-aaa_categorieTesti.py/r10761 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
 application/json
wymowa/rg2777 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/rg10878 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
dmashine-CleanupTematic.py/r10836 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
cwikt-interwiki.py/rg10901 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
t9/rg2770 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - -
 application/json
 -
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
 -
compat-basictemplatedisamblist.py/rg10907 Pywikipediabot/1.0 -
 -
botpywi-analyse_article.py/r10244 Pywikipediabot/1.0 - -
 application/json
 -
t9/rg2816 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-zzredirectyeh.py/r10582 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
refreshfeeds/rg2803 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
refreshfeeds/rg2805 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
svn-compat-fixing_redirects.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
GoogleBot - -
 text/..
pwb/rs3792 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-git-new_iw.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
('python-wikitools/1.1.1 (User:BernsteinBot)',) - -
 application/json
GoogleBot-Image/1.0 bg -
 image/..
 text/..
pywikibot-compat-replace.py/r10843 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
 application/x-www-form-urlencoded
cwikt-interwiki.py/rg10899 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
EarwigBot/0.2.dev.git487b246b (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
 application/x-www-form-urlencoded
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
CaBot Script (running on nightshade.toolserver.org) - -
 application/vnd.php.serialized
pwb/rg2864 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
erfgoedbot-categorize_images.py/r11633 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
erfgoedbot-categorize_images.py/r11633 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rs3851 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikibot-1newar25.py/rg10892 Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg10895 Pywikipediabot/1.0 - -
 application/json
 -
add_text/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
python-wikitools/1.2 (User:Cerabot) - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-git-importwdcatnl.py/r90 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.57; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
('python-wikitools/1.1.1 (User:BernsteinBot)',) -
 application/x-www-form-urlencoded
logpatrol/rg2770 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
pywikipedia-cityarticles2.py/r10866 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
 -
 text/..
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
wymowa/rg2897 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-wikidata_afb_getter.py/r11200 Pywikipediabot/1.0 - -
 application/json
My Bot - -
 text/..
svn-compat-commonscat.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
cwikt-interwiki.py/rg10899 Pywikipediabot/1.0 - -
 application/json
 application/xml
 application/x-www-form-urlencoded
redirect/rg2813 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
AdMedia bot - -
 text/..
 image/..
Screaming Frog SEO Spider/2.22 - -
 text/..
 -
 image/..
pywikipedia-reflinks.py/r11763 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/xml
 -
 application/json
pwb/rg2836 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-1newar25.py/rg10893 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
update-regular-users/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
GoogleBot fr -
 text/..
compat-gs-VM-auto-erl2.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
niki-bot - -
 text/..
 image/..
redirect/rg2809 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rg2777 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Owlin Bot v. 3.0 - -
 text/..
pywikipedia-wikidata_afb_getter.py/r11200 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Program-school_score.py/r11747 Pywikipediabot/1.0 -
 application/json
compat-tp_project.py/r10385 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 text/..
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
archiv-erl-bot/rg2834 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
gsa-crawler (Enterprise; T4-KW9PXNCULXJSR; mail address ) - -
 text/..
BotMapBeta/2.7.0.1398 CFNetwork/672.1.13 Darwin/14.0.0 de-de -
 image/..
 text/..
MediaWiki::Bot/5.005007 - -
 application/json
 -
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
pwb/rs3781 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
request-daemon/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
alswiki-title/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
replace_empty_category/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
headers/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
python-wikitools/1.2 (User:LaraBot) -
 application/json
BotProoXi fr -
 text/..
refreshfeeds/rg2815 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
gsa-crawler (Enterprise; T3-GMDVWEKT9WSQZ; mail address ) - -
 text/..
 image/..
WikiBot/0.1 - -
 text/..
LinkGraphBot/0.3 Trying to follow links. (to the past) mail address - -
 -
 application/json
datetemplates/r2403 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-redirect.py/rs10855 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
py-new_interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
 application/x-www-form-urlencoded
TestBot fr -
 text/..
 -
Innovazion Crawler/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
refreshfeeds/rg2905 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - -
 application/json
 -
 application/x-www-form-urlencoded
compat-redirect.py/rs10872 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikipedia-featured.py/rg10880 Pywikipediabot/1.0 - -
 application/json
 -
BotMapDev/2.7.0.1398 CFNetwork/672.1.13 Darwin/14.0.0 de-de -
 image/..
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
bitlybot - -
 text/..
 image/..
 -
wymowa/rg2815 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-zzcity_article.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywiki-cfd.py/rg10907 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
refreshfeeds/rg2816 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
mijnbots-ZenV.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/xml
 -
 application/json
commons-substituter.py/r11436 Pywikipediabot/1.0 - -
 application/json
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
pwb/r2618 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider,gzip(gfe) - -
 -
 text/..
py-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywiki-zzporb.py/r10870 Pywikipediabot/1.0 - -
 application/json
 -
commons-substituter.py/r11436 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
37030.07total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Apr 21, 2014 18:41
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers