Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Apr 2014 - 30 Apr 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 132,633,700 page requests (mime type text/html only!) per day are considered crawler requests, out of 647,345,130 external requests, which is 20.5%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-(compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)Accept: text/html,application/xhtmlxml,application/xml;image/png,image/jpeg,image/*;q=0.9,*/*;q=0.8 en-us;q=0.7, en;q=0.3 -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~doingfly) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxycgi1-hrd) - -
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 0.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 code.google.com/p/ldspider/wiki/Robotstext/..ldspider (url) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/p/ldspider/wiki/Robotsapplication/rdf+xmlldspider (url) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066a) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066a) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ydachao-3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~faproxyserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appengineapplication/jsonPython-urllib/1.17 AppEngine-Google; (url; appid: s~fs-lcp) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/p/ldspider/wiki/Robots-ldspider (url) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2; rv:28.0) Gecko/20100101 Firefox/28.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066d) zh-CN,zh;q=0.8 -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: suzetteklierocks) - -
 code.google.com/p/crawler4j/image/..crawler4j (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: student-pupil-help) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066d) zh-CN,zh;q=0.8 -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appengine-Python-urllib/1.17 AppEngine-Google; (url; appid: s~fs-lcp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066f) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nuraiitj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kha1989led) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vadim-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: red-arg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~projection11111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gregatr0n) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2; rv:28.0) Gecko/20100101 Firefox/28.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~certificate-check-403) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: s~certificate-check-403) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~policevan2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: predictionwrong) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066k) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: ungoodthinkful) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lil05warrior) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066f) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ahsanovic) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beansacks) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ydachao-2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~certificate-check-403) - https=1
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nhsportal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: this-is-not-what-u-think) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066b) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066k) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: raja584sekhar) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yblk-ext-proxy) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ad0066b) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: itechgiz-proxy-server) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ffffnm) zh-CN,zh;q=0.8,zh-TW;q=0.6 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zmomyanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitalhunch-freenet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: shadowtaker-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tpbitalia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~e-ddos) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.baidu.com/search/spider.html-(compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/rsd+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US https=1
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn -
 www.google.comtext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url /bot.html) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.html-(compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en https=1
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url bot.html) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1;url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,ee;q=0.8,en-gb;q=0.5,en;q=0.3 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
 www.google.com/bot.htmtext/..Mozilla/5.0 (compatible;GoogleBot/2.1;=url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 google.com/bot.htmltext/..Mozilla/5.0(comatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-gb,en;q=0.5 zero=470-01
 google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1;url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 https=1
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url bot.html) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en,no-nb,no;q=0.5 -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - https=1
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) - https=1
 search.msn.com/msnbot.htmapplication/jsonmsnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-18-generic; java 1.6.0_30; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.0ct; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_45; GMT-03:00/pt) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-60-generic; java 1.6.0_30; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-15-generic; java 1.6.0_30; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.13-1-686-pae; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (allip/any; amd64 Linux 3.13-1-amd64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.5.1.el6.x86_64; java 1.6.0_30; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-18-generic; java 1.7.0_51; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0-sabayon; java 1.7.0_51; Europe/nl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-61-generic; java 1.6.0_30; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; i386 Linux 3.13-1-686-pae; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 FreeBSD 9.2-RELEASE-p3; java 1.8.0; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.13-1-686-pae; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.13-1-686-pae; java 1.7.0_51; Europe/es) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-19-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_51; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (allip/any; amd64 Linux 3.13-1-amd64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.11.0-19-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13-1-amd64; java 1.7.0_55; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13-1-amd64; java 1.7.0_51; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-38-generic; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
commoncrawl
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 commoncrawl.org/faq/-CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 commoncrawl.org/faq/text/..CCBot/2.0 (url) - -
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 https=1
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.97~bl; url) - -
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.22.0 url - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation-WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggle-Huggle/2.1.22.0 url - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.22.0 url -
 en.wikipedia.org/wiki/Category:Algorithms/image/..Mozilla/5.0 (compatible; heritrix/1.14.4url) - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) - -
 fulltext.sblog.cz/-Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
 nodejs.orgapplication/jsonNodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
 nodejs.orgapplication/jsonNodeJsWikipediaReader/0.2 (url; mail address ) BasedOnNodeJs/0.10.23 - -
 nodejs.orgtext/..NodeJsWikipediaReader/0.2 (url; mail address ) BasedOnNodeJs/0.10.23 - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) ru -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.html-Mozilla/5.0 (iPhone; CPU iPhone OS 3_0 like Mac OS X; en-us; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) - -
naver
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 https=1
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.19 - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
allbookstores
 www.allbookstores.comapplication/jsonurl - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/-Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; newscoccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
pinterest
 pinterest.com/image/..Pinterest/0.1 url - -
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/-Pinterest/0.1 url - -
site
 www.site.com/application/jsonBot/2.0 (url) -
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-158-gf687455 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-193-g5246986 (url) en -
 scrapy.org-Scrapy/0.23.0-158-gf687455 (url) en -
 scrapy.orgtext/..Scrapy/0.22.2 (url) en -
 scrapy.orgtext/..Scrapy/0.16.3 (url) en -
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-191-ge482c6e (url) en -
 scrapy.orgtext/..Scrapy/0.22.0 (url) en -
 scrapy.orgtext/..Scrapy/0.20.0 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
wordpress
 whatownsme.wordpress.comtext/..WordPress/3.9-beta2-27714; url - -
 y35pm.wordpress.comtext/..WordPress/3.9-RC1; url - -
 y35pm.wordpress.comtext/..WordPress/3.9-beta2-27714; url - -
 y35pm.wordpress.comtext/..WordPress/3.9; url - -
 theislandjournal.wordpress.comtext/..WordPress/3.9-beta2-27714; url - -
 propelsteps.wordpress.comtext/..WordPress/3.9-RC1; url - -
 goldendragonfamily.wordpress.comtext/..WordPress/3.9-RC1; url - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
 www.archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..(compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ru -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics - url - -
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics 2.14 - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics 2.13.7 - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics - url - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - -
xbmc
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.2;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Linux; Android; 3.0.36 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (iOS; 13.0.0, Version 6.1.4 (Build 10B809); url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130527-a36a92b (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Linux; Android; 3.0.50 armv7l; url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
example
 example.orgapplication/jsonArticle-Level Metrics 2.13.7 - url - -
 example.comapplication/jsonArticle-Level Metrics - url - -
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.comapplication/jsonArticle-Level Metrics 2.13.7 - url - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.comtext/..FeedlyApp/1.0 (url) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
github
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; ChinasoSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; ChinasoSpider; url) - -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) -
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - -
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.html-Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
wesee
 www.wesee.com/en/support/bot/image/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/en/support/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
cloudflare
 www.cloudflare.com/always-onlineimage/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) ja, en -
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) - -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - -
bit
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) de
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) en
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/text/..DispensersTools (url) -
 toolserver.org/~dispenser/text/..CacheThumbs/1.4 (url) -
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot; url) - -
pooyam
 www.pooyam.com/text/..Mozilla/5.0 (compatible; PooyamBot/1.0; url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/text/..Drupal (url) -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - -
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url) -
localhost
 localhost/clumpit/bot.phptext/..Mozilla/5.0 (compatible; clumboot; url) - -
 localhost/yioop/bot.phptext/..Mozilla/5.0 (compatible; TestBot; url) - -
http:
 http://alm.example.orgapplication/jsonArticle-Level Metrics 2.14 - url - -
 http://www.goguide-directory.com.au/application/vnd.php.serializedTesting for url - -
 http://alm.example.orgapplication/jsonArticle Level Metrics - url - -
 http://alm.example.orgapplication/jsonArticle-Level Metrics - url - -
 http://www.goguide-directory.com.au/application/vnd.php.serializedurl - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
pososhok
 pososhok.eu/application/jsonPososhok/1.1 (url; mail address ) Gecko/20100101 Firefox/25.0 - -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
kinshoo
 www.kinshoo.com/bot.htmltext/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 -
 www.kinshoo.com/bot.html-KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 -
 www.kinshoo.com/bot.htmlimage/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 -
 www.kinshoo.com/bot.htmltext/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 https=1
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
 www.phonifier.com-Mozilla/5.0 (compatible; Phonifier; url) - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
 www.grapeshot.co.uk/crawler.phpimage/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
 www.proximic.com/info/spider.phpimage/..Mozilla/5.0 (compatible; proximic; url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.com-Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
trendiction
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
wikidict
 www.wikidict.detext/..url - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
 www.rcdtokyo.com/pc2m/-Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl.htmlimage/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
fueto
 fueto.comapplication/jsonFueto (url) - -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
riderswholesale
 riderswholesale.comtext/..WordPress/3.7.1; url - -
 riderswholesale.com-WordPress/3.7.1; url - -
grouphigh
 www.grouphigh.com/bot.htmltext/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
besaba
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
 www.okian.ro/application/jsonMyBot/1.0 (url) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
Anonymouse
 Anonymouse.org/text/..url (Unix) - -
 Anonymouse.org/image/..url (Unix) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
tineye
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.0.9; url ) en-us,en;q=0.5 -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.0.9; url ) - -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) en-us,en;q=0.5 -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
blogspot
 alberapps.blogspot.comtext/..TiempoBus/2.8 (url; mail address ) - -
 beh1001.blogspot.com/p/the-bot.htmltext/..Mozilla/5.0 (compatible; Beh1001/beta; url) - -
 alberapps.blogspot.com-TiempoBus/2.8 (url; mail address ) - -
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
meximas
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo-crawler; url; mail address ) - -
 text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url www.google.com/bot.html) en-US,en;q=0.5 zero=470-01
 text/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) urlGoogleBot.com Gecko/20100101 Firefox/23.0 en-us,en;q=0.5 -
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
tiscali
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.18.81 url) * -
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.18.81 url) - -
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.10.2 url) * -
nipponpaint
 nipponpaint.com.vntext/..WordPress/3.5.1; url - -
emilkirkegaard
 emilkirkegaard.dk/xkcdwiki/text/..url - -
netnewswireapp
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.12 (url) - -
servicesonrequest
 www.servicesonrequest.comtext/..User-Agent: WikiBot (url) - -
 www.servicesonrequest.com/application/jsonWikiBot(url) - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
languagetool
 community.languagetool.orgapplication/xmlurl bot, contact: mail address - -
 community.languagetool.orgtext/..url bot, contact: mail address - -
 community.languagetool.org-url bot, contact: mail address - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmltext/..psbot-image (url) - -
simplepie
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
botify
 botify.comtext/..Mozilla/5.0 (compatible; botify; url) - -
plagsniffer
 plagsniffer.de/application/jsonPlagsniffer (url) -
petisud
 www.petisud.comtext/..WordPress/3.9; url - -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
import
 import.iotext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
uni-kl
 regioguide.dfki.uni-kl.de/web/application/jsonRegioGuide/2.0 (Python; url) - -
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
rebelmouse
 rebelmouse.comimage/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.comtext/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
mytest
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
QtWeb
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
hon
 www.hon.ch/honbot/text/..HonBot/0.01 ( mail address url) - -
willmartin
 willmartin.com-WordPress/3.8.1; url - -
 willmartin.comtext/..WordPress/3.8.3; url - -
 willmartin.comtext/..WordPress/3.8.1; url - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.8 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.8 (url) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.3 (url) -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
mondowindow
 www.mondowindow.comtext/..MondoWindow (url) - -
temnos
 temnos.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
creativepulses
 creativepulses.nltext/..CreativePulses Crawler (url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
feeddemon
 www.feeddemon.com/application/xmlFeedDemon/4.5 (url; Microsoft Windows) - -
com?type=714647&fr=spigot-yhp-ch
weserv
 images.weserv.nl/image/..Mozilla/5.0 (compatible; ImageFetcher/5.5; url) -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
datoid
 datoid.cz/tag-information/application/jsonDatoid/0.9 (url; mail address ) BasedOnNette/2.0.9 - -
178074.559999973total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 7) - -
 application/vnd.php.serialized
 text/..
 -
 application/json
pwb/rg2746 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 application/pdf
py-radehh.py/r10870 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
 text/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
maj_articles_recents/rg2805 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
LinkParser/2.0 - -
 text/..
 -
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
ClueBot/2.0 - -
 application/vnd.php.serialized
 text/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
tigerbot - -
 application/json
 -
 text/..
 image/..
wikiwix-bot-3.0 - -
 text/..
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywikipedia-rad1.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
 -
ProoXiBot fr -
 text/..
 -
 application/ogg
pwb/rg2823 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
pywikibot-1newar25.py/rg10926 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/json
 -
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 text/..
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
TrueKnowledgeBot bot mail address > - -
 application/xml
 application/vnd.php.serialized
 image/..
 text/..
pywikipedia-rad1.py/rg10908 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
PythonWikipediaBot/1.0 - -
 application/json
 text/..
 -
 application/x-www-form-urlencoded
 application/xml
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN -
 text/..
 -
 application/xml
 application/ogg
pywikipedia-radeh.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
run_replace.7r/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 text/..
pywikipedia-rad1.py/r10694 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-rad1.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
Twitterbot/1.0 - -
 text/..
 image/..
 -
 application/pdf
 application/ogg
MediaWiki::Bot/5.005006 - -
 application/json
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pywikibot-1newar25.py/rg10908 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - -
 application/json
 text/..
 -
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - -
 application/json
pwb/rg2959 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
MediaWiki::Bot/3.005002 - -
 application/json
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
maj_articles_recents/rg3005 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-importwdq2.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Nutch Spider/Nutch-2.2.1 - -
 text/..
 image/..
 application/ogg
 audio/midi
pwb/rg2944 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
MediaWiki::Bot/3.2.6 - -
 application/json
 text/..
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg10930 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-1newar25.py/rg10914 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pwb/rs3943 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywiki-newar25.py/rg10901 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
pwb/rg2864 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
cis455crawler - -
 text/..
 image/..
 application/rsd+xml
 application/opensearchdescription+xml
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 -
 application/ogg
 image/..
YBot/0.1 - -
 application/vnd.php.serialized
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 -
 application/json
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 text/..
 -
 application/xml
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
 application/json
spider - -
 text/..
 application/vnd.php.serialized
 application/json
 image/..
pywikibot-1newar25.py/rg10927 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
pywikipedia-zzfatranslator.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
wiktionary-z_paginasrequeridas.py/r10529 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-wikidata_add_label_only.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
DigitalsmithsBot - -
 text/..
pywikipedia-git-wdph94.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
WikidataBot framework - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
BOT/0.1 (BOT for JCE) - -
 text/..
 application/x-www-form-urlencoded
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
pywiki-newar25.py/rg10925 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikibot-1newar25.py/rg10934 Pywikipediabot/1.0 - -
 application/json
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - -
 application/json
 -
 application/x-www-form-urlencoded
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-git-maghalesaz.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Webbot - -
 text/..
pywikipedia-git-importwdq.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-featured.py/rg10914 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
Mozilla/5.0 (Windows; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - mail address - -
 text/..
pywiki-newar25.py/rg10926 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
pywikibot-compat-commonscat.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/x-www-form-urlencoded
 application/json
 application/xml
360spider-image - -
 image/..
 text/..
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
pywikibot-1newar25.py/rg10940 Pywikipediabot/1.0 - -
 application/json
 application/xml
createHierarchy/rg2825 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rs3983 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
SentiBot www.sentibot.eu (compatible with GoogleBot) - -
 text/..
 -
maj_articles_recents/rg3012 Pywikipediabot/2.0 - -
 application/json
pywikipedia-tstwikidata.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
add/rg2916 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
createHierarchy/rg2955 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rg2916 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; SputnikBot/2.3) ru,en;q=0.1,*;q=0.01 -
 text/..
 -
PythonWikipediaBot/1.0 -
 application/json
 -
 text/..
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
ContextAd Bot 1.0 - -
 text/..
 image/..
porzucone/rg2916 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-newinterwiki-fa.py/r10334 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rs3952 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-git-maghalesaz2.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
 -
wikiparser/1 CFNetwork/673.2.1 Darwin/13.1.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 - https=1
 text/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
database_handler/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
pywikipedia-zzinter.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
cis455crawler en-US,en;q=1.0 -
 text/..
 application/ogg
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
 -
 application/pdf
 video/webm
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-1newar25.py/rg10947 Pywikipediabot/1.0 - -
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
Yeobot/1.1 Contact: mail address ) - -
 text/..
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
 -
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2916 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
pywikipedia-mdn.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pwb/rs4039 Pywikipediabot/2.0 - -
 application/json
milog_bot/1.0 ( mail address ) - -
 text/..
cwikt-interwiki.py/rg10925 Pywikipediabot/1.0 - -
 application/json
 application/xml
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10441 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pwb/rs3987 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rg2955 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
harvest_nltaxo/rg2955 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg2959 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
 application/x-www-form-urlencoded
Python27-pythonw.exe/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
Kolega2357-cosmetic_changes.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
KatBot bot.nytlabs.com> - -
 text/..
GermCrawler - -
 application/json
 application/xml
 text/..
pywikibot-1newar25.py/rg10925 Pywikipediabot/1.0 - -
 application/json
 -
mail address mail address \xE2\x80\x93 MediaWiki Tcl Bot Framework 0.5 - -
 application/x-www-form-urlencoded
German Wikipedia Broken Weblinks Bot; contact: mail address - -
 text/..
 image/..
pywiki-cfd.py/rg10926 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
pywikipedia-wikidata_add_label_only.py/r11200 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
wymowa/rg2959 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
pwb/r2772 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
coordinate_import/rs3944 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
By IdeasBot Kernel, Version:0.9.0.10 - -
 text/..
pywikipedia-new_interwiki.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikibot-compat-commonscat.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-basictemplatedisamblist.py/rg10914 Pywikipediabot/1.0 - -
 application/json
pwb/rs4030 Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
 -
pwb/rs3944 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-basictemplatedisamblist.py/rg10939 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 - https=1
 text/..
 -
pywiki-cfd.py/rg10907 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
compat-basictemplatedisamblist.py/rg10914 Pywikipediabot/1.0 -
 -
pywikipedia-zzporb-main.py/rg10908 Pywikipediabot/1.0 - -
 application/json
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
pywikibot-1newar25.py/rg10952 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.1.13 Darwin/14.0.0 en-us -
 image/..
 text/..
My Nutch Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
 image/..
 application/ogg
compat-checkimages.py/rs10905 Pywikipediabot/1.0 - -
 application/json
 -
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
compat-checkimages.py/rs10894 Pywikipediabot/1.0 - -
 -
 application/json
compat-newinterwiki-fa.py/r10334 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
pwb/rs3991 Pywikipediabot/2.0 - -
 application/json
porzucone/rg2925 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
gnd/r2916 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
replace_empty_category/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
pywikibot-1newar25.py/rg10941 Pywikipediabot/1.0 - -
 application/json
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 -
 application/json
 text/..
pywikibot-compat-commonscat.py/rg10926 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - -
 application/json
XLinkBot/1.00 - -
 text/..
 -
Mozilla/5.0 (compatible; FlappyBot; ) - -
 text/..
 image/..
239887/rg2925 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/r2669 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
TambonBot Wikibase.NET/0.1 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg2953 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
porzucone/rg2954 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-basictemplatedisamblist.py/rg10907 Pywikipediabot/1.0 -
 -
logpatrol/rg2915 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
 -
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
 application/javascript
pywikibot-compat-commonscat.py/rg10941 Pywikipediabot/1.0 - -
 application/json
 application/xml
Peachy MediaWiki Bot API Version 1.0 -
 application/vnd.php.serialized
compat-basictemplatedisamblist.py/rg10907 Pywikipediabot/1.0 - -
 application/json
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wikiparser/1 CFNetwork/673.4 Darwin/13.1.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
pywikipedia-radeh.py/rg10941 Pywikipediabot/1.0 - -
 application/json
pywiki-newar25.py/rg10941 Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-null.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzfatranslator.py/rg10941 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 -
 text/..
 image/..
t9/rg2816 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-zzredirectyeh.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
aleph-avbot.py/r11368 Pywikipediabot/1.0 -
 application/json
 text/..
 -
SurakWare MediaWiki Bot/1.0 - -
 text/..
 application/x-www-form-urlencoded
pwb/rs4053 Pywikipediabot/2.0 - -
 application/json
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 - -
 application/json
 application/xml
GoogleBot (gocrawl v0.4) - -
 text/..
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
cwikt-interwiki.py/rg10925 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
election/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 image/..
 application/ogg
 -
cwikt-interwiki.py/rg10908 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
HTMLParser/2.0 - -
 text/..
 -
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
pywikipedia-zzredirectyeh.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-linktranslator.py/rg10901 Pywikipediabot/1.0 - -
 -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - -
 application/json
pyreza-radeh2.py/r10778 Pywikipediabot/1.0 - -
 application/json
mail address - -
 application/json
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
pywikibot-1newar25.py/rg10946 Pywikipediabot/1.0 - -
 application/json
BotMapBeta/2.7.0.1398 CFNetwork/672.1.13 Darwin/14.0.0 de-de -
 image/..
 text/..
 application/rsd+xml
 application/opensearchdescription+xml
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en-US,* -
 image/..
 text/..
cis455crawler en -
 text/..
 application/ogg
recent/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzredirectyeh.py/r10694 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
German Wikipedia Dead Weblinks Checker Bot; contact: mail address - -
 text/..
pywikipedia-git-importwdq6.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
refreshfeeds/rg2916 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/rg10926 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pwb/r2954 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/rg10926 Pywikipediabot/1.0 - -
 application/json
pwb/rg2940 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
cwikt-interwiki.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
refreshfeeds/rg2959 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
wymowa/rg2953 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/rg10908 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia-zztakhasosi.py/rg10908 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) -
 application/x-www-form-urlencoded
logpatrol/rg2953 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
TripAdvisor-WikipediaGeoRequestBot/0.1 ( mail address ) - -
 application/json
compat-featured.py/rg10895 Pywikipediabot/1.0 - -
 application/json
 -
wymowa/rg2954 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
addDescOrt/r2916 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 application/xml
 -
pywikipedia-git-wdph107.py/r90 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) - -
 text/..
 -
 application/xml
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywikibot-compat-sichteroptin.py/r10880 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 text/..
 application/json
 -
pywikibot-1newar25.py/rg10910 Pywikipediabot/1.0 - -
 -
 application/json
pywikipedia-tstcat.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
pwb/rs3955 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - -
 application/json
 -
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
 application/x-www-form-urlencoded
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
BotMapDev/2.7.0.1398 CFNetwork/672.1.13 Darwin/14.0.0 de-de -
 image/..
 text/..
Erel Bot - -
 text/..
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Kolega2357-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/xml
 application/x-www-form-urlencoded
 application/json
 -
compat-basicarticledisamblist.py/rg10941 Pywikipediabot/1.0 - -
 application/json
compat-basicarticledisamblist.py/rg10908 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - -
 text/..
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
pywikipedia-zzporb-main.py/rg10908 Pywikipediabot/1.0 -
 -
COIBot/1.00 - -
 text/..
 -
compat-basicarticledisamblist.py/rg10926 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10914 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikipedia2-welcome.py/r10550 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pwb/rg2974 Pywikipediabot/2.0 - -
 application/json
ITeco bot, v0.0, mail address - -
 text/..
 -
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
pywikipedia-git-importwdq5.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywiki-linktranslator.py/rg10926 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10940 Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/rg2960 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
compat-basicarticledisamblist.py/rg10926 Pywikipediabot/1.0 -
 -
470840/rg3005 Pywikipediabot/2.0 - -
 application/json
porzucone/rg2952 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 -
 text/..
pwb/rs3981 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
porzucone/rg3011 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirectsur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pwb/rs4040 Pywikipediabot/2.0 - -
 application/json
pwb/rg2772 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rs3999 Pywikipediabot/2.0 - -
 application/json
compat-basicarticledisamblist.py/rg10908 Pywikipediabot/1.0 -
 -
pywikipedia-commonscat.py/r10786 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
porzucone/rg2966 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10930 Pywikipediabot/1.0 - -
 application/json
porzucone/rg2922 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rs3982 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
HRoestBot, de-wikipedia using pywikipedia framework -
 text/..
 application/json
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - -
 application/xml
 text/..
 -
 application/atom+xml
createHierarchy/rg2954 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
pywikibot-compat-commonscat.py/rg10939 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/rg13 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - -
 application/json
 -
COIBot/2.0 - -
 text/..
pywikipedia-zzhayati.py/rg10908 Pywikipediabot/1.0 - -
 application/json
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - -
 application/xml
 text/..
Screaming Frog SEO Spider/2.30 - -
 text/..
 image/..
 -
pywikibot-compat-commonscat.py/rg10907 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pyreza-zz_make_cat.py/r10778 Pywikipediabot/1.0 - -
 application/json
refreshfeeds/rg2954 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
python-wikitools/1.2 (User:Mr.Z-bot) - -
 application/json
 application/x-www-form-urlencoded
compat-redirect.py/rs10905 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-zzlangcat.py/rg12 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
BotMapDev/2.7.0.1398 CFNetwork/672.1.13 Darwin/13.1.0 de-de -
 image/..
 text/..
 application/opensearchdescription+xml
pywikipedia-git-wdph106.py/r90 Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
pwb/rg2863 Pywikipediabot/2.0 - -
 application/json
wAPI/1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
compat-featured.py/rg10939 Pywikipediabot/1.0 - -
 application/json
pywiki-linktranslator.py/rg10941 Pywikipediabot/1.0 - -
 application/json
archiv-erl-bot/rg2948 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-1newar25.py/rg10943 Pywikipediabot/1.0 - -
 application/json
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/rg12 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
compat-featured.py/rg10914 Pywikipediabot/1.0 -
 -
pywiki-doi.py/rg10926 Pywikipediabot/1.0 - -
 application/json
RBot/0.3 ( mail address ) -
 text/..
 image/..
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
 application/json
Projet Sport-Projetsportliste2.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
wymowa/rg2966 Pywikipediabot/2.0 - -
 application/json
wymowa/rg2925 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-redirect.py/rs10887 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
Magus Bot 1.0 - -
 text/..
 application/xml
compat-featured.py/rg10940 Pywikipediabot/1.0 - -
 application/json
User:Bilderbot (derivativeFX) by Luxo on the Toolserver / PHP -
 application/vnd.php.serialized
 text/..
run_replace.7r.4/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Innovazion Crawler/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/rg2924 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2940 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-commonscat.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
TestBot fr -
 text/..
pywikipedia-addzumra.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
Spinuf Spider - -
 text/..
pywikipedia-git-importwdq2.py/r123 Pywikipediabot/1.0 - -
 -
 application/json
pyreza-radeh.py/r10778 Pywikipediabot/1.0 - -
 application/json
Livelapbot/0.1 en-US,en -
 text/..
 image/..
 application/pdf
 -
pywikipedia-zzaddnavebox.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
MoodleBot/1.0 - -
 application/xml
 text/..
 application/vnd.php.serialized
 image/..
 -
 application/ogg
BotProoXi fr -
 text/..
pwb/rs3980 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
WPBot 1.0 - -
 text/..
 -
 application/x-www-form-urlencoded
HTMLParser/1.6 - -
 text/..
 image/..
 -
pywikipedia-zzinter.py/rg10941 Pywikipediabot/1.0 - -
 application/json
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 -
 application/json
pwb/rs4038 Pywikipediabot/2.0 - -
 application/json
pywikipedia-interwikimr.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
pywikibot-1newar25.py/rg10939 Pywikipediabot/1.0 - -
 application/json
pywikipedia-new_interwiki.py/r10866 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-featured.py/rg10947 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-new_iw14.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
refreshfeeds/rg2953 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko, Robot by TuiNu - -
 text/..
pyreza-porbinandeh2.py/r10778 Pywikipediabot/1.0 - -
 application/json
User-Agent Baiduspider - -
 text/..
 image/..
compat-checkimages.py/rs10913 Pywikipediabot/1.0 - -
 application/json
pwb/r2816 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Inlibris.com XMLBot/1.0 - -
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - -
 text/..
 application/xml
 application/json
 application/opensearchdescription+xml
svn-compat-fixing_redirects.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
BotMapBeta/2.7.0.1398 CFNetwork/672.1.13 Darwin/13.1.0 de-de -
 image/..
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
pywikipedia-zzgallery.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pwb/rg2800 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Bot\Pikosec.Com - -
 text/..
 application/opensearchdescription+xml
compat-checkimages.py/rs10920 Pywikipediabot/1.0 - -
 application/json
GoogleBot fr -
 text/..
My Bot - -
 text/..
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 text/..
 -
 application/x-www-form-urlencoded
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 text/..
 image/..
logpatrol/rg3011 Pywikipediabot/2.0 - -
 application/json
porzucone/rg2961 Pywikipediabot/2.0 - -
 application/json
CaBot Script (running on nightshade.toolserver.org) - -
 application/vnd.php.serialized
 text/..
compat-redirect.py/rs10904 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
compat-tp_project.py/r10385 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
porzucone/rg3013 Pywikipediabot/2.0 - -
 application/json
TFAdefenderbot/r2303 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg3010 Pywikipediabot/2.0 - -
 application/json
EarwigBot/0.2.dev.git700b57de (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
 application/x-www-form-urlencoded
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
porzucone/rg3012 Pywikipediabot/2.0 - -
 application/json
pwb/rs4059 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zztakhasosi.py/rg10908 Pywikipediabot/1.0 -
 -
('python-wikitools/1.1.1 (User:BernsteinBot)',) - -
 application/json
update-regular-users/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
My Nutch Spider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
cd-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
dmashine-CleanupTematic.py/r10836 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 en-us -
 image/..
 text/..
GoogleBot-Image/1.0 zh-CN,zh;q=0.8,en;q=0.6 -
 image/..
 text/..
 -
porzucone/rg3029 Pywikipediabot/2.0 - -
 application/json
wymowa/rg2961 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg3008 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
 -
 text/..
compat-checkimages.py/rs10887 Pywikipediabot/1.0 - -
 application/json
 -
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
pywikipedia-zzporb-main.py/r10694 Pywikipediabot/1.0 -
 -
datetemplates/r2403 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Python27-python.exe/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
porzucone/rg3027 Pywikipediabot/2.0 - -
 application/json
compat-welcome.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pwb/rs4057 Pywikipediabot/2.0 - -
 application/json
porzucone/rg2988 Pywikipediabot/2.0 - -
 application/json
CorenSearchBot/1.7 en libwww-perl/6.03 - -
 text/..
 application/x-www-form-urlencoded
 application/json
compat-redirect.py/rs10880 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
pywikibot-compat-commonscat.py/rg10927 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10934 Pywikipediabot/1.0 - -
 application/json
refreshfeeds/rg2925 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Kolega2357-category_redirect.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikipedia-zzfawikipatroler.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
Kolega2357-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
 -
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
articles/r2554 Pywikipediabot/2.0 - -
 application/json
botpywi-analyse_article.py/r10244 Pywikipediabot/1.0 - -
 application/json
 -
porzucone/rg3005 Pywikipediabot/2.0 - -
 application/json
pyreza-porbinandeh2.py/r10778 Pywikipediabot/1.0 -
 -
pwb/rg3005 Pywikipediabot/2.0 - -
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
wymowa/rg3031 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzhayati.py/rg10908 Pywikipediabot/1.0 -
 -
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
illustrate_wikidata/rg3029 Pywikipediabot/2.0 - -
 application/json
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
pywikibot-compat-pwb.py/r10860 Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
 application/xml
 image/..
 text/..
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
illustrate_wikidata/rg3031 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-WPP.py/r90 Pywikipediabot/1.0 - -
 application/json
request-daemon/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
python-wikitools/1.2 (User:LaraBot) -
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1; http;//www.google.com/bot.html) en-US,en;q=0.5 zero=470-01
 text/..
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1 - -
 image/..
 text/..
 -
GoogleBot-Image/1.0 bg -
 text/..
 image/..
flickrKenFielding/rg2953 Pywikipediabot/2.0 - -
 application/json
cdh-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzcity_article.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
MediaWiki::Bot/5.005007 - -
 application/json
Owlin Bot v. 3.0 - -
 text/..
WikiBot/0.1 - -
 text/..
wymowa/rg3006 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3027 Pywikipediabot/2.0 - -
 application/json
gsa-crawler (Enterprise; T4-KW9PXNCULXJSR; mail address ) - -
 text/..
pywikipedia-reupload_tsh.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia-git-importwdq.py/r123 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
UniversalFeedParser/5.1.1 https://code.google.com/p/feedparser/ - -
 application/xml
 text/..
AnomieBOT 1.0 (BrokenRedirectDeleter; see [[User:AnomieBOT III]]) - -
 application/json
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 text/..
pywikipedia-zzporb-main.py/r10694 Pywikipediabot/1.0 - -
 application/json
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
wymowa/rg3004 Pywikipediabot/2.0 - -
 application/json
pywiki-delete.py/rg10925 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-featured.py/rg10926 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; SputnikImageBot/2.2) - -
 image/..
 text/..
add_text/rg2911 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pyreza-zz_make_cat.py/r10778 Pywikipediabot/1.0 -
 -
Wikibot/2.1.1 CFNetwork/672.1.14 Darwin/14.0.0 en-us -
 image/..
 text/..
Arkbot/0.1 alpha -
 application/json
zitat/rg2781 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3029 Pywikipediabot/2.0 - -
 application/json
logpatrol/rg3005 Pywikipediabot/2.0 - -
 application/json
bitlybot - -
 text/..
 image/..
 -
pywikipedia-zzlangcat.py/rg13 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.1; ru-ru; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 ru-RU, en-US -
 image/..
 -
 text/..
createHierarchy/rg2956 Pywikipediabot/2.0 - -
 application/json
wymowa/rg2973 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3013 Pywikipediabot/2.0 - -
 application/json
pywikipedia2-checkimages.py/r10550 Pywikipediabot/1.0 - -
 application/json
 -
pyiw-redirect.py/r11641 Pywikipediabot/1.0 - -
 application/json
Py-importwdq.py/rs10905 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-pwb.py/rg10914 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 -
 application/xml
 image/..
YisouSpider - -
 text/..
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
illustrate_wikidata/rg3009 Pywikipediabot/2.0 - -
 application/json
39070.85total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Thu, May 15, 2014 12:54
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers