Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Nov 2014 - 30 Nov 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 145,612,600 page requests (mime type text/html only!) per day are considered crawler requests, out of 678,356,330 external requests, which is 21.5%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1;php=zend
 www.bing.com/bingbot.htm-Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - php=zend
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1;php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - https=1;php=zend
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6001-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile8-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~cgiproxy3-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~guidesites-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy4-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~lookingforwe-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~usethisproxy-hrd) - php=zend
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk9-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~overviewa-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~goodersearch-hrd) - php=zend
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - https=1;php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~appcodes-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy-py-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~payonlyproxy-hrd) - php=zend
 www.bing.com/bingbot.htmimage/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxycgi1-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~front-pages-hrd) - php=zend
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 code.google.com/p/crawler4j/text/..crawler4j (url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=hhvm
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - php=zend
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~jnhanproxy) - php=zend
 code.google.com/p/crawler4j/text/..crawler4j (url) - php=hhvm
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - php=zend
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - php=hhvm
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - php=zend
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - php=zend
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jayjudah1) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - php=zend
 code.google.com/p/crawler4j/image/..crawler4j (url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=hhvm
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0 AppEngine-Google; (url; appid: s~pagedaili7) en-US php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ungoodthinkful) - php=zend
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp2) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - php=zend
 code.google.com/appenginetext/..firefox AppEngine-Google; (url; appid: s~positronews) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: jayjudah1) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - php=zend
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - https=1;php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - php=zend
 code.google.com/appenginetext/..Python-urllib/2.7 AppEngine-Google; (url; appid: s~servewibe) - php=zend
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - php=zend
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~imgdaili3) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - php=zend
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - php=zend
 code.google.com/p/ldspider/wiki/Robotstext/..ldspider (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~harunakaze) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~argentina-country) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~bayatcybercity) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - php=zend
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - php=zend
 code.google.com/p/gwtwiki/text/..JavaWikipediaAPI/3.1-SNAPSHOT url - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - php=zend
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: carlos-web-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lukaviy419) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~e-ddos) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps002) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - php=zend
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: chronickg) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn03) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yes9987) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~no8lancelot) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~223-p4) zh-CN,zh;q=0.8,en;q=0.6 php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaxminds) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vibhoreindia) - php=zend
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~jnhanproxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - php=zend
 www.google.com/adsbot.htmltext/..AdsBot-Google (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~449-lsw3) zh-CN,zh;q=0.8,en;q=0.6 php=zend
 code.google.com/appengineapplication/jsonPython-urllib/2.7 AppEngine-Google; (url; appid: s~context-engine) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chronickg) - php=zend
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - php=zend
 code.google.com/appenginetext/..Mozilla/5.0 AppEngine-Google; (url; appid: s~ax-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp5) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demowaiy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: caralho-voador) - php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zoxy-1) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: goeverysite) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - php=zend
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - php=zend
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-us,zh;q=0.9 https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=hhvm
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US php=zend
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - https=1;php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8,bn;q=0.6 php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-us,zh;q=0.9 https=1;php=hhvm
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 php=zend
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8,bn;q=0.6 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b3 - php=zend
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=hhvm
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US, en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=hhvm
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.htmlapplication/oggMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5.0 - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg php=hhvm
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ru,en-us;q=0.7,en;q=0.7 php=zend
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en,* php=zend
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1;php=zend
 www.baidu.com/search/spi_der.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 php=hhvm
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - php=hhvm
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - php=hhvm
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US php=zend
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=hhvm
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US https=1;php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1;php=hhvm
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en,* php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.5b5 - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url),gzip(gfe) - php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - php=hhvm
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en,* php=zend
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - php=hhvm
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - php=zend
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - php=hhvm
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - php=zend
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - php=hhvm
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* php=zend
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - php=zend
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1;php=zend
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-US,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* php=hhvm
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - php=hhvm
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X; ja-jp) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Y!J-ASR/1.0 fetcher (Y!J-AGENT; url)) en-US,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-US,* php=zend
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1;php=hhvm
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X; ja-jp) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Y!J-ASR/1.0 fetcher (Y!J-AGENT; url)) en,* -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Mozilla/5.0 (compatible; Y!J-ASR/1.0 fetcher (Y!J-AGENT; url)) - php=zend
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=hhvm
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) tr, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1;php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/vnd.php.serializedMozilla/5.0 (compatible; YandexBot/3.0; url) ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3 php=zend
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - php=zend
 yandex.com/botstext/..198.143.34.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU -
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=hhvm
 yandex.com/botstext/..198.143.47.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - php=zend
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru,en;q=0.8 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) en-us, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru-RU php=zend
 yandex.com/botstext/..108.162.254.94 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4 php=hhvm
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - php=zend
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - php=hhvm
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=hhvm
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - https=1;php=zend
cibra
 cibra.de/text/..CiBra Data Collector (url) - php=zend
 cibra.de/text/..CiBra Data Collector (url) - php=hhvm
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - php=zend
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - php=zend
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - php=hhvm
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - php=hhvm
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
xip
 bestbuyindustrialscientific.shophddvsbestdeal.com.v4oqzx.xip.io/vanessausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.bestoffertoybabykidssale.com.7fsoqa.xip.io/daisyusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.brodybestbuy.com.tmql2q.xip.io/LikeBuytext/..WordPress/4.0; url - php=zend
 www.guideforbuyonline.com.uz9zjm.xip.io/Thebesttext/..WordPress/4.0; url - php=zend
 www.lookforbestpricebuy.com.1nskhpp.xip.io/Buycheaptext/..WordPress/4.0; url - php=zend
 morebestpricepccomponent.bestusaguide.com.fh72uq.xip.io/reviewbuytext/..WordPress/4.0; url - php=zend
 bestbuybeautysale.bestofferdesktopsale.com.1utce2a.xip.io/arleneusabestbuy2utext/..WordPress/4.0; url - php=zend
 bigsavetoykids.marshallbestbuy.com.11hutrm.xip.io/NowDealtext/..WordPress/4.0; url - php=zend
 www.shopbestbuysale.com.1qa706l.xip.io/caroleusabestbuy2utext/..WordPress/4.0; url - php=zend
 hotbestdealtoolshomeimprovement.bestdealblackfridaycybermonday.com.1d1wraq.xip.io/belindausabestbuy2utext/..WordPress/4.0; url - php=zend
 bestoffergrocerygourmetfood.bestofferdesktopsale.com.1utce2a.xip.io/arleneusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.elibestbuy.com.1ug2bhp.xip.io/ProvenGoodtext/..WordPress/4.0; url - php=zend
 bestbabytoy.bilalbestbuy.com.1c826uq.xip.io/NowSavetext/..WordPress/4.0; url - php=zend
 www.syedbestbuy.com.1xz5b59.xip.io/ProvenGettext/..WordPress/4.0; url - php=zend
 bestbuyreplacementparts.topbestpricehomeaudio.com.1bk4n8t.xip.io/jessieusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.jonahbestbuy.com.1nskhpp.xip.io/ProvenBesttext/..WordPress/4.0; url - php=zend
 bestpricekitchendining.bestpricetoygamessale.com.17s8oqa.xip.io/miriamusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.timothybestbuy.com.9nieya.xip.io/SecretBesttext/..WordPress/4.0; url - php=zend
 camerathebestonline.guideforbuyonline.com.uz9zjm.xip.io/Thebesttext/..WordPress/4.0; url - php=zend
 bestpriceskincare.bestdealshopelectronics.com.1ewp225.xip.io/courtneyusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.guideforusagetnow.com.eaf2y5.xip.io/Searchbesttext/..WordPress/4.0; url - php=zend
 www.shopelectroniccomputer.com.1qa706l.xip.io/jennieusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.janbestbuy.com.1owenjh.xip.io/SecretFindtext/..WordPress/4.0; url - php=zend
 bestpricepatiolawngarden.bestoffertoybabykidssale.com.7fsoqa.xip.io/daisyusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.justinbestbuy.com.1owenjh.xip.io/EasyViewtext/..WordPress/4.0; url - php=zend
 www.leonardobestbuy.com.1c826uq.xip.io/LikeBesttext/..WordPress/4.0; url - php=zend
 bestofferexterioraccessories.hotnewbestbuylaptop.com.v4oqzx.xip.io/robertausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.hotbestbuyelectronics.com.1dvzr5e.xip.io/jackieusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.milobestbuy.com.p3z8vh.xip.io/ProvenBuytext/..WordPress/4.0; url - php=zend
 www.marshallbestbuy.com.11hutrm.xip.io/NowDealtext/..WordPress/4.0; url - php=zend
 www.bestdealshopelectronics.com.1ewp225.xip.io/courtneyusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestpricebabytoddlertoys.bestpricetoygamessale.com.17s8oqa.xip.io/miriamusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestdealcarvehicleelectronics.hotnewbestbuylaptop.com.v4oqzx.xip.io/robertausabestbuy2utext/..WordPress/4.0; url - php=zend
 shopbestdealhomedecor.shopbestbuysale.com.1qa706l.xip.io/caroleusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestcomputeraccessories.thelowpriceonlineusa.com.p3z8vh.xip.io/Onlinebuytext/..WordPress/4.0; url - php=zend
 www.teddybestbuy.com.sipeeq.xip.io/OnlineReviewtext/..WordPress/4.0; url - php=zend
 www.guidethebestbuy.com.1owenjh.xip.io/cheapchecktext/..WordPress/4.0; url - php=zend
 bestdealhealthpersonalcare.hotbestbuyelectronics.com.1dvzr5e.xip.io/jackieusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.bestusaguide.com.fh72uq.xip.io/reviewbuytext/..WordPress/4.0; url - php=zend
 bestpriceoccupationalhealth.hotbestbuycamera.com.1bk4n8t.xip.io/eileenusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.jessebestbuy.com.1owenjh.xip.io/YouGettext/..WordPress/4.0; url - php=zend
 www.xavierbestbuy.com.uz9zjm.xip.io/ProvenMoretext/..WordPress/4.0; url - php=zend
 www.checkforcheapusa.com.1qtibal.xip.io/checkbesttext/..WordPress/4.0; url - php=zend
 www.hassanbestbuy.com.11bacvh.xip.io/ResultChecktext/..WordPress/4.0; url - php=zend
 bestpricehaircare.bestofferdesktopsale.com.1utce2a.xip.io/arleneusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.niallbestbuy.com.6ntjjm.xip.io/YouDealtext/..WordPress/4.0; url - php=zend
 www.mohamedbestbuy.com.1xb1po2.xip.io/FindReviewtext/..WordPress/4.0; url - php=zend
 bestbuycampinghiking.bestdealblackfridaycybermonday.com.1d1wraq.xip.io/belindausabestbuy2utext/..WordPress/4.0; url - php=zend
 bestbuyinterioraccessories.shophddvsbestdeal.com.v4oqzx.xip.io/vanessausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.adrianbestbuy.com.1owenjh.xip.io/LikeChecktext/..WordPress/4.0; url - php=zend
 www.jordanbestbuy.com.1qtibal.xip.io/LikeGettext/..WordPress/4.0; url - php=zend
 www.moreusabestbuy.com.u3p4ml.xip.io/Buytodaytext/..WordPress/4.0; url - php=zend
 www.hotbestbuycamera.com.1bk4n8t.xip.io/eileenusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestofferlightingceilingfans.bestpricetoygamessale.com.17s8oqa.xip.io/miriamusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestpriceboatingwatersports.bestdealblackfridaycybermonday.com.1d1wraq.xip.io/belindausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.bilalbestbuy.com.1c826uq.xip.io/NowSavetext/..WordPress/4.0; url - php=zend
 www.jasonbestbuy.com.eaf2y5.xip.io/NowSaletext/..WordPress/4.0; url - php=zend
 www.laytonbestbuy.com.nzzgrx.xip.io/NowBesttext/..WordPress/4.0; url - php=zend
 bestbuymotorcyclepowersports.hotbestbuycamera.com.1bk4n8t.xip.io/eileenusabestbuy2u-WordPress/4.0; url - php=zend
 bestdealhouseholdsupplies.hotbestbuyelectronics.com.1dvzr5e.xip.io/jackieusabestbuy2utext/..WordPress/4.0; url - php=zend
 topbestdealbedding.shopelectroniccomputer.com.1qa706l.xip.io/jennieusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestcarelectronics.lookforbestpricebuy.com.1nskhpp.xip.io/Buycheaptext/..WordPress/4.0; url - php=zend
 www.frankbestbuy.com.fh72uq.xip.io/EasySaletext/..WordPress/4.0; url - php=zend
 www.pricecheckerusa.com.6iqg8d.xip.io/Cheapreviewtext/..WordPress/4.0; url - php=zend
 www.bestpriceshopsalenow.com.1ewp225.xip.io/irmausabestbuy2utext/..WordPress/4.0; url - php=zend
 bestpricevitaminsdietarysupplements.bestdealshopelectronics.com.1ewp225.xip.io/courtneyusabestbuy2utext/..WordPress/4.0; url - php=zend
 hotbestdealhandtools.shopbestbuysale.com.1qa706l.xip.io/caroleusabestbuy2utext/..WordPress/4.0; url - php=zend
 bigsavemorecamera.getthebestpriceusa.com.1xz5b59.xip.io/Buyguidetext/..WordPress/4.0; url - php=zend
 bestdealfurnituresale.shopbestbuysale.com.1qa706l.xip.io/caroleusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.lukasbestbuy.com.6ntjjm.xip.io/ResultFindtext/..WordPress/4.0; url - php=zend
 www.usaonlinecheapguide.com.1c826uq.xip.io/getbesttext/..WordPress/4.0; url - php=zend
 www.rafaelbestbuy.com.11hutrm.xip.io/GuideViewtext/..WordPress/4.0; url - php=zend
 www.travisbestbuy.com.fh72uq.xip.io/ResultDirecttext/..WordPress/4.0; url - php=zend
 bestpricepowerhandtools.bestoffertoybabykidssale.com.7fsoqa.xip.io/daisyusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.lincolnbestbuy.com.ruazzh.xip.io/Guidesbuytext/..WordPress/4.0; url - php=zend
 www.mylesbestbuy.com.ruazzh.xip.io/LikeOnlinetext/..WordPress/4.0; url - php=zend
 www.lovebestbuyusa.com.1ug2bhp.xip.io/Cheapnowtext/..WordPress/4.0; url - php=zend
 bestdollsaccessories.stevenbestbuy.com.ruazzh.xip.io/SecretSaletext/..WordPress/4.0; url - php=zend
 www.peterbestbuy.com.61p3jm.xip.io/SeeReviewtext/..WordPress/4.0; url - php=zend
 bestcheapmore.bestforusaonline.com.ruazzh.xip.io/bigsavetext/..WordPress/4.0; url - php=zend
 www.kacperbestbuy.com.9j1sq5.xip.io/MoreSaletext/..WordPress/4.0; url - php=zend
 www.spencerbestbuy.com.9nieya.xip.io/YouSavetext/..WordPress/4.0; url - php=zend
 www.mckenziebestbuy.com.93htma.xip.io/Moreviewtext/..WordPress/4.0; url - php=zend
 www.flynnbestbuy.com.17p3tt9.xip.io/ProvenSavetext/..WordPress/4.0; url - php=zend
 www.conorbestbuy.com.9nieya.xip.io/GuideChecktext/..WordPress/4.0; url - php=zend
 bigguideonlinedeals.laytonbestbuy.com.nzzgrx.xip.io/NowBesttext/..WordPress/4.0; url - php=zend
 bestbuyappliancessale.bestpriceshopsalenow.com.1ewp225.xip.io/irmausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.emmanuelbestbuy.com.1c826uq.xip.io/BuyAleksanderBesttext/..WordPress/4.0; url - php=zend
 bestpricepersonalcare.bestdealshopelectronics.com.1ewp225.xip.io/courtneyusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.arjunbestbuy.com.fh72uq.xip.io/GuideSaletext/..WordPress/4.0; url - php=zend
 bestbuyartscraftssewing.bestpriceshopsalenow.com.1ewp225.xip.io/irmausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.stevenbestbuy.com.ruazzh.xip.io/SecretSaletext/..WordPress/4.0; url - php=zend
 www.morebestbuyinusa.com.11hutrm.xip.io/findbesttext/..WordPress/4.0; url - php=zend
 www.abdullahbestbuy.com.u3p4ml.xip.io/NowCheaptext/..WordPress/4.0; url - php=zend
 bestbuyperformanceparts.topbestpricehomeaudio.com.1bk4n8t.xip.io/jessieusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestbuybathroomfixtures.bestpriceshopsalenow.com.1ewp225.xip.io/irmausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.jaibestbuy.com.1qtibal.xip.io/GuideGettext/..WordPress/4.0; url - php=zend
 bestbuyrvparts.topbestpricehomeaudio.com.1bk4n8t.xip.io/jessieusabestbuy2utext/..WordPress/4.0; url - php=zend
 guideforbigsaveusa.jonahbestbuy.com.1nskhpp.xip.io/ProvenBesttext/..WordPress/4.0; url - php=zend
 bestdealmedicalsuppliesequipment.hotbestbuyelectronics.com.1dvzr5e.xip.io/jackieusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.declanbestbuy.com.11bacvh.xip.io/YouCheaptext/..WordPress/4.0; url - php=zend
 guidetobuyusa.elibestbuy.com.1ug2bhp.xip.io/ProvenGoodtext/..WordPress/4.0; url - php=zend
 www.ahmedbestbuy.com.61p3jm.xip.io/MoreBuytext/..WordPress/4.0; url - php=zend
 www.jadenbestbuy.com.ruazzh.xip.io/BuyFabianBesttext/..WordPress/4.0; url - php=zend
 www.anthonybestbuy.com.1m3jlil.xip.io/GetReviewtext/..WordPress/4.0; url - php=zend
 desktopsthebestusa.pricecheckerusa.com.6iqg8d.xip.io/Cheapreviewtext/..WordPress/4.0; url - php=zend
 www.robinbestbuy.com.fh72uq.xip.io/SecretBuytext/..WordPress/4.0; url - php=zend
 bestinstrumentaccessoriesusa.onlinethenestseller.com.tmql2q.xip.io/reviewtodaytext/..WordPress/4.0; url - php=zend
 bestbuytoolsequipment.topbestpricehomeaudio.com.1bk4n8t.xip.io/jessieusabestbuy2utext/..WordPress/4.0; url - php=zend
 hotbestbuydeal.shopelectroniccomputer.com.1qa706l.xip.io/jennieusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.ryleybestbuy.com.9nieya.xip.io/LikeSaletext/..WordPress/4.0; url - php=zend
 www.dannybestbuy.com.ruazzh.xip.io/MoreFindtext/..WordPress/4.0; url - php=zend
 bestelectronicsusa.checkforcheapusa.com.1qtibal.xip.io/checkbesttext/..WordPress/4.0; url - php=zend
 bestbuybathsale.bestpriceshopsalenow.com.1ewp225.xip.io/irmausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.albiebestbuy.com.11hutrm.xip.io/LikeReviewtext/..WordPress/4.0; url - php=zend
 bestkidshobbies.janbestbuy.com.1owenjh.xip.io/SecretFindtext/..WordPress/4.0; url - php=zend
 www.szymonbestbuy.com.1c826uq.xip.io/GuideBesttext/..WordPress/4.0; url - php=zend
 bestpricepowertoolaccessories.bestoffertoybabykidssale.com.7fsoqa.xip.io/daisyusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestmp3playeraccessories.usaonlinecheapguide.com.1c826uq.xip.io/getbesttext/..WordPress/4.0; url - php=zend
 bestpricepetsupplies.bestoffertoybabykidssale.com.7fsoqa.xip.io/daisyusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.bestdealblackfridaycybermonday.com.1d1wraq.xip.io/belindausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.getthebestpriceusa.com.1xz5b59.xip.io/Buyguidetext/..WordPress/4.0; url - php=zend
 bestdealhealthcare.hotbestbuyelectronics.com.1dvzr5e.xip.io/jackieusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.bestbuynowusa.com.11bacvh.xip.io/cybersearchtext/..WordPress/4.0; url - php=zend
 bestdealhardware.shopbestbuysale.com.1qa706l.xip.io/caroleusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.scottbestbuy.com.9nieya.xip.io/EasyBesttext/..WordPress/4.0; url - php=zend
 hotbestdealbuildingsupplies.shopelectroniccomputer.com.1qa706l.xip.io/jennieusabestbuy2utext/..WordPress/4.0; url - php=zend
 justbestprice.flynnbestbuy.com.17p3tt9.xip.io/ProvenSavetext/..WordPress/4.0; url - php=zend
 www.romanbestbuy.com.1m3jlil.xip.io/MoreDealtext/..WordPress/4.0; url - php=zend
 bestlaptopsmoresaveusa.searchforbestbuyus.com.17p3tt9.xip.io/Savetodaytext/..WordPress/4.0; url - php=zend
 getbestpricelaptops.lovebestbuyusa.com.1ug2bhp.xip.io/Cheapnowtext/..WordPress/4.0; url - php=zend
 bestactionfigures.declanbestbuy.com.11bacvh.xip.io/YouCheaptext/..WordPress/4.0; url - php=zend
 www.thebestdealsusa.com.nzzgrx.xip.io/Bestreviewtext/..WordPress/4.0; url - php=zend
 www.rubenbestbuy.com.sipeeq.xip.io/MoreGettext/..WordPress/4.0; url - php=zend
 bestkidsgames.timothybestbuy.com.9nieya.xip.io/SecretBesttext/..WordPress/4.0; url - php=zend
 findbestpricehere.bestusaguide.com.fh72uq.xip.io/reviewbuytext/..WordPress/4.0; url - php=zend
 www.topbestpricehomeaudio.com.1bk4n8t.xip.io/jessieusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestmoreonline.cyberbestsellerusa.com.6ntjjm.xip.io/lovebesttext/..WordPress/4.0; url - php=zend
 www.joeybestbuy.com.1owenjh.xip.io/GuideMoretext/..WordPress/4.0; url - php=zend
 www.milesbestbuy.com.tmql2q.xip.io/YouBesttext/..WordPress/4.0; url - php=zend
 gobestonline.abdullahbestbuy.com.u3p4ml.xip.io/NowCheaptext/..WordPress/4.0; url - php=zend
 www.marleybestbuy.com.93htma.xip.io/CheapReviewstext/..WordPress/4.0; url - php=zend
 www.maximilianbestbuy.com.1xb1po2.xip.io/moresavepricetext/..WordPress/4.0; url - php=zend
 www.cyberbestsellerusa.com.6ntjjm.xip.io/lovebesttext/..WordPress/4.0; url - php=zend
 www.juniorbestbuy.com.1qtibal.xip.io/EasyChecktext/..WordPress/4.0; url - php=zend
 www.bestpricetoygamessale.com.17s8oqa.xip.io/miriamusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestfeedingstuff.niallbestbuy.com.6ntjjm.xip.io/YouDealtext/..WordPress/4.0; url - php=zend
 www.bestofferdesktopsale.com.1utce2a.xip.io/arleneusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.jaidenbestbuy.com.1jmkyp9.xip.io/MoreDiscounttext/..WordPress/4.0; url - php=zend
 www.caidenbestbuy.com.11hutrm.xip.io/EasyGuidetext/..WordPress/4.0; url - php=zend
 www.declanbestbuy.com.1sle8fm.xip.io/MoreReviewtext/..WordPress/4.0; url - php=zend
 www.martinbestbuy.com.tmql2q.xip.io/EasyCheaptext/..WordPress/4.0; url - php=zend
 bigsave4shop.pricecheckerusa.com.6iqg8d.xip.io/Cheapreviewtext/..WordPress/4.0; url - php=zend
 bestgrownuptoys.bestsellers4usa.com.tmql2q.xip.io/SecretMoretext/..WordPress/4.0; url - php=zend
 www.riobestbuy.com.6iqg8d.xip.io/NowBuytext/..WordPress/4.0; url - php=zend
 bestkidsnursery.dillonbestbuy.com.1qtibal.xip.io/NowViewtext/..WordPress/4.0; url - php=zend
 www.brooklynbestbuy.com.tmql2q.xip.io/GuideBiztext/..WordPress/4.0; url - php=zend
 www.thelowpriceonlineusa.com.p3z8vh.xip.io/Onlinebuytext/..WordPress/4.0; url - php=zend
 bestonlinethebest.searchforbestbuyus.com.17p3tt9.xip.io/Savetodaytext/..WordPress/4.0; url - php=zend
 www.hotshopbestpricesale.com.1bk4n8t.xip.io/terriusabestbuy2utext/..WordPress/4.0; url - php=zend
 checkfordealsusa.xavierbestbuy.com.uz9zjm.xip.io/ProvenMoretext/..WordPress/4.0; url - php=zend
 www.raphaelbestbuy.com.6ntjjm.xip.io/EasyBuytext/..WordPress/4.0; url - php=zend
 likeshopbestprice.cybershopchecker.com.9nieya.xip.io/morebesttext/..WordPress/4.0; url - php=zend
 bigsavepriceusadeals.guideforbuyonline.com.uz9zjm.xip.io/Thebesttext/..WordPress/4.0; url - php=zend
 www.bestsellers4usa.com.tmql2q.xip.io/SecretMoretext/..WordPress/4.0; url - php=zend
 lookforbestsallerusa.getthebestpriceusa.com.1xz5b59.xip.io/Buyguidetext/..WordPress/4.0; url - php=zend
 www.carterbestbuy.com.1owenjh.xip.io/BuySamiBesttext/..WordPress/4.0; url - php=zend
 www.beaubestbuy.com.1qtibal.xip.io/YouPricetext/..WordPress/4.0; url - php=zend
 www.freddybestbuy.com.1qtibal.xip.io/BuyKrishBesttext/..WordPress/4.0; url - php=zend
 goodpriceus.teddybestbuy.com.sipeeq.xip.io/OnlineReviewtext/..WordPress/4.0; url - php=zend
 www.buygunholsters.us.1kjn776.xip.io/buyGunHolsterstext/..WordPress/4.0.1; url - php=zend
 getthebestusa.lovebestbuyusa.com.1ug2bhp.xip.io/Cheapnowtext/..WordPress/4.0; url - php=zend
 bigsaveonline.jasonbestbuy.com.eaf2y5.xip.io/NowSaletext/..WordPress/4.0; url - php=zend
 www.eliasbestbuy.com.1c826uq.xip.io/EasyOnlinetext/..WordPress/4.0; url - php=zend
 www.shophddvsbestdeal.com.v4oqzx.xip.io/vanessausabestbuy2utext/..WordPress/4.0; url - php=zend
 great4online.guideforusagetnow.com.eaf2y5.xip.io/Searchbesttext/..WordPress/4.0; url - php=zend
 morethebest.marleybestbuy.com.93htma.xip.io/CheapReviewstext/..WordPress/4.0; url - php=zend
 www.hamzabestbuy.com.11hutrm.xip.io/YouSaletext/..WordPress/4.0; url - php=zend
 bestbuytacticaldutydeal.hotshoppingbestbuysale.v4oqzx.xip.io/hollyusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.dillonbestbuy.com.1qtibal.xip.io/NowViewtext/..WordPress/4.0; url - php=zend
 computercomponent.cyberbestsellerusa.com.6ntjjm.xip.io/lovebesttext/..WordPress/4.0; url - php=zend
 dealsthebest.bilalbestbuy.com.1c826uq.xip.io/NowSavetext/..WordPress/4.0; url - php=zend
 www.kadenbestbuy.com.ruazzh.xip.io/BuyPrinceBesttext/..WordPress/4.0; url - php=zend
 bestblogitemsreviews.thelowpriceonlineusa.com.p3z8vh.xip.io/Onlinebuytext/..WordPress/4.0; url - php=zend
 findbigsave.thebestdealsusa.com.nzzgrx.xip.io/Bestreviewtext/..WordPress/4.0; url - php=zend
 www.hotnewbestbuylaptop.com.v4oqzx.xip.io/robertausabestbuy2utext/..WordPress/4.0; url - php=zend
 bestpricenaturalorganic.bestdealshopelectronics.com.1ewp225.xip.io/courtneyusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.reggiebestbuy.com.11bacvh.xip.io/MoreOnlinetext/..WordPress/4.0; url - php=zend
 www.cybershopchecker.com.9nieya.xip.io/morebesttext/..WordPress/4.0; url - php=zend
 www.searchforbestbuyus.com.17p3tt9.xip.io/Savetodaytext/..WordPress/4.0; url - php=zend
 bestdealexercisefitness.bestbuyhdtvsusa.com.6blvsi.xip.io/hattieusabestbuy2utext/..WordPress/4.0; url - php=zend
 bestcomponentpc.bestforusaonline.com.ruazzh.xip.io/bigsavetext/..WordPress/4.0; url - php=zend
 www.rufusbestbuy.com.11bacvh.xip.io/EasySavetext/..WordPress/4.0; url - php=zend
 www.cyberbestbuysaleonline.com.1dvzr5e.xip.io/marciausabestbuy2utext/..WordPress/4.0; url - php=zend
 www.marcusbestbuy.com.hm5zu5.xip.io/MoreBettertext/..WordPress/4.0; url - php=zend
 www.zachbestbuy.com.1sle8fm.xip.io/DiscoverMoretext/..WordPress/4.0; url - php=zend
 www.mateuszbestbuy.com.11bacvh.xip.io/GuideSavetext/..WordPress/4.0; url - php=zend
 moresave4usa.moreusabestbuy.com.u3p4ml.xip.io/Buytodaytext/..WordPress/4.0; url - php=zend
 www.zakbestbuy.com.9j1sq5.xip.io/CheckReviewtext/..WordPress/4.0; url - php=zend
 www.ismailbestbuy.com.ruazzh.xip.io/EasyDealtext/..WordPress/4.0; url - php=zend
 www.oskarbestbuy.com.fh72uq.xip.io/MoreSearchtext/..WordPress/4.0; url - php=zend
 bestdealkitchenbathfixtures.bestpricetoygamessale.com.17s8oqa.xip.io/miriamusabestbuy2utext/..WordPress/4.0; url - php=zend
 www.colebestbuy.com.1jmkyp9.xip.io/LoveReviewtext/..WordPress/4.0; url - php=zend
 bestshoppinglowprice.guidethebestbuy.com.1owenjh.xip.io/cheapchecktext/..WordPress/4.0; url - php=zend
 bestbigsavemoreusa.lookforbestpricebuy.com.1nskhpp.xip.io/Buycheaptext/..WordPress/4.0; url - php=zend
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=zend
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=hhvm
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - php=zend
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=zend
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/RSS/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=zend
msn
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - php=zend
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - php=zend
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - php=hhvm
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - php=hhvm
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - php=hhvm
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - php=hhvm
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
naver
 help.naver.com/robots/application/jsonMozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
 help.naver.com/robots/image/..Mozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/application/jsonMozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=hhvm
 help.naver.com/robots/application/jsonMozilla/5.0 (compatible; Yeti/1.1; url) en-US php=zend
 help.naver.com/robots/text/..Mozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
 help.naver.com/robots/application/jsonMozilla/5.0 (compatible; Yeti/1.1; url) - php=zend
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en php=hhvm
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en php=hhvm
 the.echonest.com/reader/application/xmlnestReader/0.4 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/application/xmlnestReader/0.4 (discovery; url; reader at echonest.com) en php=hhvm
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - php=zend
 wikipedia.orange.fr/text/..API/1.0 (url) - php=hhvm
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - php=zend
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - php=zend
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/MusikAnimal url en-US,* php=zend
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Vitor_Mazuco url pt-BR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Winner_42 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/I_dream_of_horses url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Becky_Sayles url en-US,* php=zend
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.26.0 url - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/HeicoH url de-DE,en,* php=zend
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBotapplication/jsonOrlodrimBot/1.0 (url) - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/wtmitchell url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/PauloEduardo url pt-BR,en,* php=zend
 ko.wikipedia.orgapplication/jsonurl - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Donner60 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Rrburke url en-CA,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/KylieTastic url en-GB,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stickee url en-AU,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Spuk968 url de-DE,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/haminoon url en-NZ,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Altsprachenfreund url de-DE,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/NGPriest url en-AU,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Marcos_dias_de_oliveira url pt-BR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Isha url es-AR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Manslay url ru-RU,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/nonsenseferret url en-GB,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/falcon8765 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Avono url en-GB,*
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/OWAIS_NAEEM url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Lugia2453 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Leon_saudanha url pt-BR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/9shi url zh-HK,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/demize url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Regi51 url de-DE,en,* php=zend
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn php=hhvm
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - php=zend
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - php=hhvm
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..web spider/4.0(url) - php=zend
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url) zh-cn -
quickiwiki
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - php=hhvm
 www.quickiwiki.comapplication/jsonQuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
commoncrawl
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - php=hhvm
 www.text/..GoogleBot/2.X (urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - php=hhvm
 www.text/..Mozilla/5.0 (compatible; GoogleBot/2.1 urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) en php=zend
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - php=zend
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - php=hhvm
wikiwand
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=hhvm
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=hhvm
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - php=zend
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - php=zend
 tools.wmflabs.org/geohacktext/..Geohack (url) - php=hhvm
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - php=hhvm
 tools.wmflabs.org/kmlexporttext/..url libwww-perl/6.03 - php=zend
 tools.wmflabs.org/geocommons/text/..url - php=zend
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 php=zend
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 php=hhvm
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - php=zend
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: go2-gg) en-us;q=0.7,en;q=0.3 php=zend
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: go2-vn) en-us;q=0.7,en;q=0.3 php=zend
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - php=zend
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - php=zend
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - php=hhvm
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - php=hhvm
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs php=zend
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) cs php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs php=hhvm
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) cs php=hhvm
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs php=hhvm
 fulltext.sblog.cz/image/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 php=hhvm
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - php=zend
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - php=hhvm
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - php=zend
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - php=zend
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.29 - php=zend
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.0.0 (url) PHP/5.5.18 ja php=zend
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.19 - -
007ac9
 crawler.007ac9.net/text/..Mozilla/5.0 (compatible; 007ac9 Crawler; url) - php=zend
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - php=zend
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - php=zend
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - php=hhvm
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - php=hhvm
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.24.4 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-216-g44cbbec (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-232-ged84231 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.22.2 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.18.1 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.24.4 (url) en php=hhvm
 scrapy.orgtext/..Scrapy/0.25.0-232-ged84231 (url) en php=hhvm
 scrapy.orgtext/..Scrapy/0.24.4 (url) en https=1;php=zend
 scrapy.orgtext/..Scrapy/0.24.2 (url) en php=zend
localhost
 localhostapplication/jsonLagotto 3.6.3 - url - php=zend
 localhostapplication/jsonLagotto 3.6.3 - url - php=hhvm
yacy
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.17.2-1-ARCH; java 1.7.0_71; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmlimage/..yacybot (/global; amd64 Linux 3.17.2-1-ARCH; java 1.7.0_71; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-39-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.7.10-1.40-desktop; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.0-24-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Linux 3.13.0-39-generic; java 1.7.0_65; Europe/fr) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-39-generic; java 1.7.0_65; Europe/fr) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.17.4-1-ARCH; java 1.7.0_71; Asia/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 8 6.2; java 1.8.0_11; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-32-generic; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Linux 3.17.4; java 1.8.0_25; Europe/ru) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-40-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.17.0-031700-generic; java 1.7.0_65; Asia/ja) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; x86 Windows 7 6.1; java 1.7.0_71; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.5.0-44-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.17.4; java 1.8.0_25; Europe/ru) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.17.4-1-ARCH; java 1.7.0_71; Asia/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-ovz-el-alt122; java 1.7.0_09-icedtea; GMT/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/nl) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - php=zend
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; Please Name Your robot; url) - php=zend
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - php=hhvm
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; Please Name Your robot; url) - php=hhvm
plos
 alm.plos.orgapplication/jsonLagotto 3.9.1 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.7 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.9.3 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.9.1 - url - php=hhvm
 alm.plos.orgapplication/jsonLagotto 3.9.3 - url - php=hhvm
 sfo-stage-alm01.int.plos.orgapplication/jsonLagotto 3.9.1 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.8 - url - php=zend
 sfo-stage-alm01.int.plos.orgapplication/jsonLagotto 3.7 - url - php=zend
 sfo-stage-alm01.int.plos.orgapplication/jsonLagotto 3.9.1 - url - php=hhvm
etao
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - php=zend
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - php=hhvm
wordpress
 goldendragon1971.wordpress.comtext/..WordPress/4.1-alpha-20141011; url - php=zend
 cuthulan.wordpress.comtext/..WordPress/4.1-alpha-20141011; url - php=zend
 peripluscd.wordpress.comtext/..WordPress/4.1-alpha-20141011; url - php=zend
 hoangkimlong.wordpress.comtext/..WordPress/4.1-alpha-20141011; url - php=zend
 wildanrenaldi.wordpress.comtext/..WordPress/4.1-alpha-30055; url - php=zend
flipboard
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 php=zend
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 php=hhvm
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - php=zend
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - php=zend
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - php=hhvm
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - php=zend
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - php=hhvm
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.98~bl; url) - php=zend
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.98~bl; url) - php=hhvm
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot-SA/0.97; url) - php=zend
pagefreezer
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 php=zend
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 php=hhvm
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-us php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - php=hhvm
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-us php=hhvm
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-gb php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-us
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address ar php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address ru php=zend
yodao
 www.yodao.com/help/webmaster/spider/application/json(compatible; YodaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 php=zend
 www.yodao.com/help/webmaster/spider/application/json(compatible; YodaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 php=hhvm
http:
 http://pkp-alm.lib.sfu.caapplication/jsonArticle-Level Metrics 3.6 - url - php=zend
 http://pkp-alm.lib.sfu.caapplication/jsonArticle-Level Metrics 3.6 - url - php=hhvm
xbmc
 www.xbmc.orgtext/..XBMC/12.4 Git:20140322-b134a7d (Linux; Unknown; 3.0.101-MX armv7l; url) - php=zend
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1; url) - -
 xbmc.org-XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.1.10-gd1a8ddd armv7l; url) - php=zend
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.3;WOW64;Win64;x64; url) - -
 xbmc.orgtext/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.1.10-gd1a8ddd armv7l; url) - php=zend
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.0.36 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140831-a1505b1 (Linux; Android; 3.10.33 armv7l; url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=zend
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=zend
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=hhvm
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=hhvm
example
 www.example.com/text/..wikidrain/1.0 (url) - php=zend
 www.example.com/text/..wikidrain/1.0 (url) - php=hhvm
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - php=zend
 example.com/MyCoolToolPage/text/..User-Agent: MyCoolTool (url) - php=zend
 example.com/MyCoolTool/application/jsonMyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - php=zend
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - php=zend
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - php=zend
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - php=hhvm
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - php=zend
wesee
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PageBot (url) - php=zend
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - php=zend
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - php=hhvm
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - php=zend
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - php=hhvm
 www.feedly.com/fetcher.htmltext/..Feedly/1.0 (url; like FeedFetcher-Google) - php=zend
recuweb
 recuweb.comapplication/jsonurl - php=zend
 recuweb.comapplication/jsonurl en-us,en;q=0.5 php=zend
 recuweb.comapplication/jsonurl en-us,en;q=0.5 php=hhvm
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en php=zend
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - php=zend
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en php=hhvm
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - php=hhvm
 www.majestic12.co.uk/bot.php?image/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
drupal
 drupal.org/application/jsonDrupal (url) - php=zend
 drupal.org/text/..Drupal (url) - php=zend
 drupal.org/application/jsonDrupal (url) - php=hhvm
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..Drupal (url) - php=hhvm
test
 test.co.kr/text/..xxxxxxxx/Nutch-2.2.1 (url) - php=zend
 test.co.kr/text/..xxxxxxxx/Nutch-2.2.1 (url) - php=hhvm
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 php=zend
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 php=hhvm
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * php=zend
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * php=hhvm
dhcp
 bestonlineusa.dhcp.io/checkbestsellertext/..WordPress/4.0; url - php=zend
riddler
 riddler.io/abouttext/..Riddler (url) - php=zend
 riddler.io/about.htmltext/..Riddler (url) - php=zend
 riddler.io/abouttext/..Riddler (url) - php=hhvm
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - php=zend
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - php=hhvm
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - php=zend
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - php=hhvm
 www.grapeshot.co.uk/crawler.phpimage/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
crazywebcrawler
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.1, url - php=zend
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.0, url - php=zend
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.1, url - php=hhvm
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de-DE, de;q=0.9, dsb-DE;q=0.3, hsb-DE;q=0.2 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de-DE, de;q=0.9, dsb-DE;q=0.3, hsb-DE;q=0.2 php=hhvm
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - php=zend
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - php=zend
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - php=zend
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; url) - php=zend
tineye
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - php=zend
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - php=hhvm
searchingmotion
 searchingmotion.com/text/..SMcrawler (url) - php=zend
 searchingmotion.com/image/..SMcrawler (url) - -
github
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 php=zend
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - php=zend
 wiki.github.com/bixo/bixo/bixocrawlertext/..Mozilla/5.0 (compatible; pub-crawler; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - php=zend
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 php=hhvm
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - php=zend
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - php=zend
uplabs0
 alm-1.uplabs0.comapplication/jsonLagotto 3.9.1 - url -
 alm-1.uplabs0.comapplication/jsonArticle-Level Metrics 3.3.8 - url -
netvibes
 www.netvibes.comtext/..Netvibes (url) - php=zend
 www.netvibes.comapplication/xmlNetvibes (url) - php=zend
 www.netvibes.comapplication/jsonNetvibes (url) en-us php=zend
 www.netvibes.comtext/..Netvibes (url) - php=hhvm
okian
 www.okian.ro/text/..MyBot/1.0 (url) - php=zend
 www.okian.ro/application/jsonMyBot/1.0 (url) - php=zend
 www.okian.ro/text/..MyBot/1.0 (url) - php=hhvm
 www.okian.ro/application/jsonMyBot/1.0 (url) - php=hhvm
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 php=zend
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 php=hhvm
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - php=zend
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - php=hhvm
zeerch
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - php=zend
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - php=zend
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - php=hhvm
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - php=hhvm
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=zend
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=hhvm
datasift
 datasift.com/bot.htmltext/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - php=zend
 datasift.com/bot.htmlimage/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - -
 datasift.com/bot.htmltext/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - php=hhvm
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - php=zend
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - php=hhvm
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=zend
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=hhvm
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - php=zend
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - php=hhvm
unifiedlayer
 50-87-201-32.unifiedlayer.com/CeciliaFindtext/..WordPress/4.0; url - php=zend
 50-87-201-249.unifiedlayer.com/AliviaOffertext/..WordPress/4.0; url - php=zend
 173-254-99-209.unifiedlayer.com/ScarlettGaintext/..WordPress/4.0; url - php=zend
 173-254-124-63.unifiedlayer.com/AshlynUnlocktext/..WordPress/4.0; url - php=zend
 173-254-124-63.unifiedlayer.com/AshlynUnlock/cheapreplacementpartstext/..WordPress/4.0; url - php=zend
 173-254-124-63.unifiedlayer.com/AshlynUnlock/findreplacementpartstext/..WordPress/4.0; url - php=zend
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - php=zend
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - php=hhvm
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - php=zend
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - php=hhvm
grouphigh
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - php=zend
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - php=hhvm
 www.grouphigh.com/application/xmlMozilla/5.0 (compatible; GroupHigh/1.0; url) - php=zend
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.4.0; url ) en-us,en;q=0.5 php=zend
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.3.0; url ) en-us,en;q=0.5 php=zend
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.4.0; url ) - php=zend
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.3.0; url ) - php=zend
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.4.0; url ) en-us,en;q=0.5 php=hhvm
draigweb
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - php=zend
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - php=zend
egerin
 www.egerin.comtext/..crawler (url); Kurdish search engine ku php=zend
 www.egerin.comtext/..crawler (url); Kurdish search engine ku php=hhvm
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - php=zend
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - php=hhvm
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - php=zend
bne
 www.bne.es/es/LaBNE/ArchivoWeb/AvisoWebmasters/index.htmltext/..Mozilla/5.0 (compatible; heritrix/1.12.1 url) - php=zend
 www.bne.es/es/LaBNE/ArchivoWeb/AvisoWebmasters/index.htmlimage/..Mozilla/5.0 (compatible; heritrix/1.12.1 url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - php=zend
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - php=hhvm
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - php=zend
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - php=hhvm
 www.saciol.com/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - php=zend
mysite
 www.mysite.com/application/vnd.php.serializedMyBot/1.0 (url) - php=zend
 www.mysite.com/application/jsonMyBot/1.0 (url) - php=zend
 www.mysite.com/text/..MyBot/1.0 (url) - php=zend
 www.mysite.com/botinfoapplication/vnd.php.serializedMozilla/5.0 (compatible; MyCoolBot/1.0; url) - php=zend
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - php=zend
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
leiki
 www.leiki.comtext/..Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; url) - php=zend
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 php=zend
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
wikidict
 www.wikidict.detext/..url - php=zend
 www.wikidict.detext/..url - php=hhvm
pinterest
 www.pinterest.com/text/..Pinterest/0.2 (url) - php=zend
 www.pinterest.com/image/..Pinterest/0.2 (url) - -
 pinterest.com/image/..Pinterest/0.1 url - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * php=zend
igrec
 www.igrec.ca/projectstext/..Wikitionary Text Parser 0.2 (url) - php=zend
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 php=zend
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - php=zend
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - php=zend
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
friedemann-vogel
 friedemann-vogel.de/forschung/konfliktlinguistik-20application/jsonTalksCrawler/2.0 (url) - php=zend
 friedemann-vogel.de/forschung/konfliktlinguistik-20application/jsonTalksCrawler/2.0 (url) - php=hhvm
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) php=zend
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) it php=zend
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en php=zend
 bsurprised.com/text/..BSurprised WikiBox 0.1 (url) en php=zend
xovibot
 www.xovibot.net/text/..Mozilla/5.0 (compatible; XoviBot/2.0; url) - php=zend
 www.xovibot.net/text/..Mozilla/5.0 (compatible; XoviBot/2.0; url) - php=hhvm
sputnik
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 php=zend
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 php=hhvm
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - php=zend
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - php=zend
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - php=zend
lagotto
 percona.lagotto.ioapplication/jsonLagotto 3.8 - url - php=zend
testomato
 www.testomato.com/testomatobottext/..Testomatobot/1.0 (Linux x86_64; url) minicrawler/2.8~alpha en-US,en,cs;q=0.01 php=zend
 www.testomato.com/testomatobottext/..Testomatobot/1.0 (Linux x86_64; url) minicrawler/3.0~alpha en-US,en,cs;q=0.01 php=zend
 www.testomato.com/testomatobottext/..Testomatobot/1.0 (Linux x86_64; url) minicrawler/2.7.11 en-US,en,cs;q=0.01 php=zend
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - php=zend
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - php=zend
netarkivet
 netarkivet.dk/webcrawlerimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawlertext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=zend
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=zend
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - php=zend
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - php=zend
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
newworldmodel
 newworldmodel.com/lotterytext/..WordPress/4.0; url - php=zend
 newworldmodel.com/netflixtext/..WordPress/4.0; url - php=zend
 newworldmodel.com/hardwood-arm-chairtext/..WordPress/4.0; url - php=zend
 newworldmodel.com/water-deliverytext/..WordPress/4.0; url - php=zend
ovh
 vps110150.ovh.net/bilatu/bot.phptext/..Mozilla/5.0 (compatible; Bilatu.eus; url) -
 vps110150.ovh.net/bilatu/bot.phptext/..Mozilla/5.0 (compatible; Bilatu.eus; url) - php=zend
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - php=zend
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - php=zend
ac
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - php=zend
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - php=zend
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - php=zend
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - php=zend
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) - -
edu
 pjwstk.edu.pltext/..Wiki Controversial/0.1; url ; mail address -
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - php=zend
 www.swust.edu.cntext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=zend
uni-leipzig
 corpora.informatik.uni-leipzig.de/crawler_faq.htmltext/..Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV url) - php=zend
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - php=zend
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - php=zend
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
yourschool
nip
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - php=zend
simplepie
great-deal-spot
 great-deal-spot.comtext/..WordPress/4.0; url - php=zend
 great-deal-spot.comtext/..WordPress/4.0.1; url - php=zend
metamagazine
 metamagazine.comtext/..WordPress/3.9.2; url -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - php=zend
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.14 (url) - php=zend
 tt-rss.org/application/xmlTiny Tiny RSS/1.13 (url) - php=zend
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - php=zend
freewpwebsitecreation
 freewpwebsitecreation.comtext/..WordPress/4.0; url - php=zend
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - php=zend
perfectmarket
 www.perfectmarket.comtext/..Mozilla/5.0 (compatible; PerfectMarketKWTbot/1.0 url) - php=zend
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
innercitylist
 classified.innercitylist.comtext/..WordPress/4.0; url - php=zend
 atlanta.innercitylist.comtext/..WordPress/4.0; url - php=zend
 atlanta.innercitylist.comtext/..WordPress/4.0.1; url - php=zend
sciencecard
 dryad.sciencecard.orgapplication/jsonArticle-Level Metrics 3.6.3 - url - php=zend
 dryad.sciencecard.orgapplication/jsonArticle-Level Metrics 3.6.3 - url - php=hhvm
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - php=zend
 www.thinglink.com/help/ThinglinkImageBottext/..Elmer, the Thinglink ImageBot (url) - php=zend
 www.thinglink.com/help/ThinglinkImageBotimage/..Elmer, the Thinglink ImageBot (url) - -
cdlib
 webarchives.cdlib.org/p/webmasterstext/..Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 url) - php=zend
bigdatacorp
 ecommerce.bigdatacorp.com.br/faq.aspxtext/..Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; url) AppleWebKit/535.1 KHTML Chrome/13.0.782.107 Safari/535.1 pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
idmarch
 www.idmarch.org/bot.htmltext/..Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; url) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
dozentenmarkt
 dozentenmarkt.de/DozBotapplication/jsonDozBot/0.1 (url; mail address ) - php=zend
nipponpaint
 nipponpaint.com.vntext/..WordPress/3.5.1; url - php=zend
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - php=zend
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - php=zend
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) -
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - php=zend
208
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - php=zend
inagist
 inagist.comtext/..InAGist URL Resolver (url) en-us,en;q=0.5 php=zend
site-shot
 www.site-shot.com/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Chromium/31.0.1650.63 Site-Shot/2.1 (url) Safari/534.34 en-US,* -
 www.site-shot.com/text/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Chromium/31.0.1650.63 Site-Shot/2.1 (url) Safari/534.34 en-US,* php=zend
diffbot
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 php=zend
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) - php=zend
image2play
 image2play.com/image/..Mozilla/5.0 (compatible; Image2play/0.1; url) - -
xxx
 www.xxx.comtext/..spider/Nutch-1.5.1 (spider; url) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
v--u
 v--u.com/colon-detoxtext/..WordPress/4.0; url - php=zend
 v--u.com/green-coffee-beantext/..WordPress/4.0; url - php=zend
bestonlinesongs
 www.bestonlinesongs.com/text/..MyBot/1.0 (url) - php=zend
best-auto-ins
 best-auto-ins.comtext/..WordPress/4.0; url - php=zend
 best-auto-ins.com/auto-repairtext/..WordPress/4.0; url - php=zend
queryseeker
 queryseeker.com/bot.htmltext/..QuerySeekerSpider ( url ) - php=zend
gorgor
 gorgor.irtext/..hivaBot/hivaBot-1.0 (Iranian Search Engine bot; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 gorgor.irapplication/jsonhivaBot/hivaBot-1.0 (Iranian Search Engine bot; url; mail address ) - php=zend
blogspot
 alberapps.blogspot.comtext/..TiempoBus/3.1.4 (url; mail address ) - php=zend
livelap
 site.livelap.com/crawlertext/..LivelapBot/0.2 (url) en-US,en php=zend
 site.livelap.com/crawlerimage/..LivelapBot/0.2 (url) en-US,en -
zmescience
 www.zmescience.comtext/..WordPress/4.0; url - php=zend
 www.zmescience.comtext/..WordPress/4.0.1; url - php=zend
anexhome
 anexhome.com/lotterytext/..WordPress/4.0; url - php=zend
 anexhome.com/iphone-WordPress/4.0; url - php=zend
owlin
 www.owlin.com/bottext/..Owlin Bot v. 4.5.1 (url) - php=zend
knowly
 www.knowly.net/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - php=zend
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 php=zend
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address ) - php=zend
temnos
 temnos.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 php=zend
scarecrow
 www.scarecrow.com/application/jsonScarecrow/1.0 (url; mail address ) - php=zend
thecardb
 thecardb.comtext/..theCarDB/1.1 (url; mail address ) - php=zend
QtWeb
hon
 www.hon.ch/honbot/text/..HonBot/0.01 ( mail address url) - -
 www.hon.ch/honbot/text/..HonBot/0.01 ( mail address url) - php=zend
creativepulses
 creativepulses.nltext/..CreativePulses Crawler (url) - php=zend
 creativepulses.nlapplication/jsonCreativePulses Crawler (url) - php=zend
psu
 citeseerx.ist.psu.edu/text/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-2014-11-14T15:29:34Z url) - php=zend
 citeseerx.ist.psu.edu/image/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-2014-11-14T15:29:34Z url) - -
themself
 www.themself.orgtext/..WordPress/4.0.1; url - php=zend
 www.themself.orgtext/..WordPress/4.0.1; url - php=hhvm
crowsnest
 www.crowsnest.tv/text/..Crowsnest/0.5 (url) - php=zend
194056.299999981total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/ogg
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - php=zend
 application/vnd.php.serialized
 text/..
 application/json
MediaWikiCrawler-Google/2.0 ( mail address ) - php=zend
 text/..
wikiwix-bot-3.0 - php=zend
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
LinkParser/2.0 - php=zend
 text/..
py-radehh.py/rg10975 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
php wikibot classes - php=zend
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN php=zend
 text/..
 application/xml
ClueBot/2.0 - php=zend
 application/vnd.php.serialized
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
GoogleBot-Image/1.0 - php=zend
 text/..
 image/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - php=zend
 application/json
 text/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
welcome (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
 text/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - php=hhvm
 application/vnd.php.serialized
 text/..
 application/json
maj_articles_recents (wikipedia:fr; User:ZéroBot) Pywikibot/2.0b2 (g4249) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
tigerbot - php=zend
 application/json
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
Mozilla/5.0 (compatible; OrangeBot/2.0; mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 php=zend
 text/..
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - php=zend
 application/json
 text/..
pwb/rg3113 Pywikipediabot/2.0 - php=zend
 application/json
 text/..
AniBot/0.9 php/curl - php=zend
 application/vnd.php.serialized
MediaWiki::Bot/5.005006 - php=zend
 application/json
Twitterbot/1.0 - php=zend
 text/..
 application/json
MediaWiki::Bot/3.2.6 - php=zend
 application/json
 text/..
pywikipedia-radeh.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
milog_bot/1.0 ( mail address ) - php=zend
 text/..
MediaWikiCrawler-Google/2.0 ( mail address ) - php=hhvm
 text/..
LinkParser/2.0 - php=hhvm
 text/..
DotNetWikiBot/3.11 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
DigitalsmithsBot - php=zend
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm
 text/..
 application/xml
images (wikipedia:pl) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
py-radehh.py/rg10975 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
UAA-UAA.py/r10417 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
spider - php=zend
 text/..
 application/json
 application/vnd.php.serialized
php wikibot classes - php=hhvm
 application/vnd.php.serialized
 text/..
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - php=zend
 text/..
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - php=zend
 application/json
pywikipedia-newinterwiki-fa.py/rg11077 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - php=hhvm
 application/json
interwiki (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
PythonWikipediaBot/1.0 - php=zend
 application/json
 text/..
pywikipedia-git-desc_bot6.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-git-desc_bot5.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
ContextAd Bot 1.0 - php=zend
 text/..
rk-crawler en-US,en;q=0.8 php=zend
 text/..
pywikipedia-zzfatranslator.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
strucr.com crawler 3.25.402 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=zend
 text/..
Owlin Bot v. 3.0 - php=zend
 text/..
 application/xml
DotNetWikiBot/3.11 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
GoogleBot-Image/1.0 - php=zend
 text/..
 application/json
 image/..
wikiwix-bot-3.0 - php=hhvm
 text/..
MediaWiki::Bot/3.005002 - php=zend
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - php=hhvm
 application/json
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
ClueBot/2.0 - php=hhvm
 application/vnd.php.serialized
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - php=hhvm
 application/json
WorldOfMusicBetaBot/1.0 - php=zend
 text/..
HTMLParser/2.0 - php=zend
 text/..
renametemp (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
inflectionof (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* php=zend
 text/..
 image/..
 application/json
cat_finder-core (wikipedia:fa; User:Rezabot) Pywikibot/2.0b2 (g4466) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
 text/..
WikidataBot framework - php=zend
 application/json
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - php=zend
 text/..
 application/xml
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; php=zend
 text/..
pywikibot-1newar25.py/rg11098 Pywikipediabot/1.0 - php=zend
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
 application/xml
python-wikitools/1.2 (User:DickensBot) -
 application/json
 text/..
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - php=zend
 text/..
welcome (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
cat_finder-core (wikidata:wikidata; User:Rezabot) Pywikibot/2.0b2 (g4466) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
wymowa (commons:commons; User:Alkamid) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
run_replace.7r.02/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - php=zend
 application/json
compat-redirect.py/rs11063 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
 application/xml
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN php=hhvm
 text/..
 application/xml
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
iw2data/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - php=hhvm
 application/json
pywikibot-1newar25.py/rg11094 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
GjirafaSpider - php=zend
 text/..
 application/xml
Spiderbot/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
pywikibot-1newar25.py/rg11090 Pywikipediabot/1.0 - php=zend
 application/json
run_replace.7r.03/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
HTMLParser/1.6 - php=zend
 text/..
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
pywikibot-1newar25.py/rg11095 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
MyCuteBot/0.1 - php=zend
 text/..
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - php=zend
 application/json
SPIupdater-globalfunc.py/r10417 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - php=zend
 application/json
Twitterbot/1.0 - -
 image/..
 text/..
 application/pdf
 application/ogg
 video/webm
 audio/webm
run (wikipedia:fr) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.8.final.0 - php=zend
 application/json
DotNetWikiBot/3.12 (Unix 3.2.0.59; ) - php=zend
 text/..
GermCrawler - php=zend
 application/json
 text/..
 application/xml
node-crawler/0.2.5 - php=zend
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.60; ) - php=zend
 text/..
 application/xml
compat-checkimages.py/rs11063 Pywikipediabot/1.0 - php=zend
 application/json
Screaming Frog SEO Spider/2.55 - php=zend
 text/..
 application/xml
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b2 (3) httplib2/0.9 Python/2.7.8.final.0 - php=zend
 application/json
 text/..
GoogleBot (gocrawl v0.4) - php=zend
 text/..
wiktionaryInterwiki/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
User-Agent Baiduspider - php=zend
 text/..
 image/..
compat-label_bot.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
Webbot - php=zend
 text/..
AnomieBOT/1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - php=hhvm
 application/json
maj_articles_recents (wikipedia:fr; User:ZéroBot) Pywikibot/2.0b2 (g4249) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
Portale (wikipedia:it; User:TauerBot) Pywikibot/2.0b2 httplib2/0.7.4 Python/2.7.3.final.0 - php=zend
 application/json
compat-basictemplatedisamblist.py/rg11090 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-git-wdlabel.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
Bot - php=zend
 text/..
pywikipedia-git-sectionBoten.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - php=hhvm
 application/json
 text/..
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
GoogleBot-Image/1.0 - https=1;php=zend
 text/..
compat-redirect.py/rs11076 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
 application/xml
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
 application/xml
python-wikitools/1.2 (User:Irclogbot) - php=zend
 application/json
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - php=zend
 text/..
 application/json
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - php=hhvm
 application/json
 text/..
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlBot) - php=zend
 text/..
mrmetadata (wikipedia:en) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
compat-checkimages.py/rs11076 Pywikipediabot/1.0 - php=zend
 application/json
refreshfeeds (wikinews:ru; User:AKBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
German Wikipedia Broken Weblinks Bot; contact: mail address - php=zend
 text/..
pwb/rg3009 Pywikipediabot/2.0 - php=zend
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
py-navbox.py/rg10975 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
python-wikitools/1.3 (User:Mr.Z-bot) - php=zend
 application/json
GoogleBot-Image/1.0 - php=hhvm
 text/..
 image/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
CorenSearchBot/1.7 en libwww-perl/6.05 - php=zend
 text/..
AnomieBOT/1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - php=hhvm
 application/json
compat-desc_bot.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
AniBot/0.9 php/curl - php=hhvm
 application/vnd.php.serialized
DotNetWikiBot/3.11 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm
 text/..
tigerbot - php=hhvm
 application/json
 text/..
sport (wikipedia:fr; User:NaggoBot) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
claimit (wikidata:wikidata; User:BotMultichill) Pywikibot/2.0b2 (g4457) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-git-desc_bot6.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
bot-07-fix-language-mappings (wikidata:wikidata; User:AkkakkBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - php=zend
 text/..
 application/xml
AnomieBOT/1.0 (TemplateSubster; see [[User:AnomieBOT]]) - php=hhvm
 application/json
geolocation (wikipedia:en; User:BotMultichill) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.2.final.0 - php=zend
 application/json
pwb/rg3113 Pywikipediabot/2.0 - php=hhvm
 application/json
SearchBot/Nutch-1.9 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
 application/xml
MediaWiki::Bot/5.005006 - php=hhvm
 application/json
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-radeh.py/rg11084 Pywikipediabot/1.0 - php=hhvm
 application/json
BotKung/4.0 - php=zend
 text/..
 application/xml
compat-basicarticledisamblist.py/rg11090 Pywikipediabot/1.0 - php=zend
 application/json
ko_missing2 (wikipedia:en; User:Chobot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - php=zend
 text/..
Goalkeeperbot(User:Beetstra)/1.0 - php=zend
 text/..
run_pagefromfile.5/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us -
 image/..
YisouSpider - php=zend
 application/json
 text/..
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - php=zend
 text/..
cat_finder-core (wikidata:wikidata; User:Rezabot) Pywikibot/2.0b2 (g4582) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-git-ul_wd.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
run_pagefromfile.4/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
cat_finder-core (wikipedia:fa; User:Rezabot) Pywikibot/2.0b2 (g4582) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
gsa-crawler (Enterprise; T3-RC5FESB3QYSGY; mail address ) - php=zend
 text/..
RBot/0.3 ( mail address ) -
 application/json
 image/..
 text/..
MediaWiki::Bot/3.2.6 - php=hhvm
 application/json
claimit (wikipedia:en; User:BotMultichill) Pywikibot/2.0b2 (g4457) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
michiel_copylabel (wikidata:wikidata; User:RobotMichiel1972) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
Pastec bot -
 image/..
 text/..
recent/r3183 Pywikibot/2.0 - php=zend
 application/json
wymowa (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
XLinkBot/1.00 - php=zend
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 php=zend
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
MediaWikiCrawler-Google - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
mussol-mussol.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
ifd-hoobot (wikidata:wikidata; User:PLbot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - php=zend
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
BOT/0.1 (BOT for JCE) - php=zend
 text/..
MediaWiki::Bot/5.005007 - php=zend
 application/json
DigitalsmithsBot - php=hhvm
 text/..
pywikipedia-git-del_new.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
images (wikipedia:es) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/3.12 (Unix 3.2.0.59; ) - php=hhvm
 text/..
run (wikipedia:fr) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.8.final.0 - php=hhvm
 application/json
cat_finder-core (wikipedia:fa; User:Rezabot) Pywikibot/2.0b2 (g4348) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
SineBot/1.5.19(User:SineBot) - php=zend
 application/vnd.php.serialized
 text/..
strucr.com crawler 3.25.402 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=hhvm
 text/..
SentiBot www.sentibot.eu (compatible with GoogleBot) - php=zend
 text/..
pywikibot-1newar25.py/rg11101 Pywikipediabot/1.0 - php=zend
 application/json
TestBot/0.1 - php=zend
 application/json
FP (commons:commons; User:SteinsplitterBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
Omnilibbot/1.0 en, fr,ro,* php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - php=hhvm
 application/json
welcome-c (wikipedia:fr; User:Loveless) Pywikibot/2.0b2 (g4566) httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
pywikibot-compat-dobrodoslice.py/rg11089 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (compatible; OrangeBot/2.0; mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 php=hhvm
 text/..
DotNetWikiBot/2.92 (Microsoft Windows NT 6.1.7600.0; ) - php=zend
 text/..
 application/xml
mrmetadata (wikipedia:ru) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
a (wikipedia:mr; User:निनावी) Pywikibot/2.0b2 httplib2/0.9 Python/3.4.2.final.0 - php=zend
 application/json
GoogleBot/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
mrmetadata (wikipedia:de) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
 text/..
images (wikipedia:pl) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
UAA-UAA.py/r10417 Pywikipediabot/1.0 - php=hhvm
 application/json
Answersbot - php=zend
 text/..
My Nutch Spider/Nutch-1.9 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - php=zend
 text/..
nullref (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - php=zend
 application/json
Twitterbot/1.0 - php=hhvm
 text/..
run_replace.7r.04/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - php=hhvm
 text/..
Peachy MediaWiki Bot API Version 1.0 - php=zend
 application/vnd.php.serialized
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
pywikipedia-zzinter.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-newinterwiki-fa.py/rg11077 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
SurakWare MediaWiki Bot/1.0 - php=zend
 text/..
 application/xml
pywikipedia-zzpurging.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
run_replace.7r.05/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
wiktionaryInterwiki/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
pywikipedia-git-com4.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-git-desc_bot5.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
 text/..
Facebot/1.0 -
 text/..
DotNetWikiBot/3.10 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - php=zend
 text/..
AnomieBOT/1.0 (BAGBot; see [[User:AnomieBOT]]) - php=hhvm
 application/json
 text/..
a (wikipedia:hi; User:निनावी) Pywikibot/2.0b2 httplib2/0.9 Python/3.4.2.final.0 - php=zend
 application/json
 text/..
TicketsBot/0.1 - php=zend
 text/..
interwiki (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
Yeobot/1.1 Contact: mail address ) - php=zend
 text/..
pywikipedia-zzfatranslator.py/rg11084 Pywikipediabot/1.0 - php=hhvm
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - php=zend
 text/..
 application/xml
interwiki (wikinews:en) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
cat_finder-core (wikidata:wikidata; User:Rezabot) Pywikibot/2.0b2 (g4348) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
MediaWiki::Bot/3.1.6 (User:SporkBot) - php=zend
 application/json
mrmetadata (wikipedia:it) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywiki-cfd.py/rg11032 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
wiktionaryInterwiki (wiktionary:mg; User:Bot-Jagwar) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
robospider/1.0 - php=zend
 text/..
MediaWiki::Bot/3.005002 - php=hhvm
 application/json
InfolinksSpider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
updateHuman/rg3599 Pywikibot/2.0 - php=zend
 application/json
Mozilla/5.0 (Apibot Browser) - -
 image/..
 text/..
 application/ogg
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
Egerin Spider 1.0 - php=zend
 text/..
Nutch Spider/Nutch-2.2.1 - php=zend
 text/..
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
touch (wikipedia:ar; User:ZkBot) Pywikibot/2.0b2 (g4601) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.3; ru-ru; Spider Build/JLS36C) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 ru-RU, en-US -
 image/..
run (wikipedia:fr) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
diacritics_redirects (wikipedia:ro; User:Strainubot) Pywikibot/2.0b2 (g4458) httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
pywikipedia-zzredirectyeh.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikibot-1newar25.py/rg11098 Pywikipediabot/1.0 - php=hhvm
 application/json
t9/rg2816 Pywikipediabot/2.0 - php=zend
 application/json
Webwiki Search Engine Bot - www.webwiki.de - php=zend
 text/..
wikispider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
inflectionof (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
mrmetadata (wikipedia:id) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
WorldOfMusicBetaBot/1.0 - php=hhvm
 text/..
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
Bot at mysite.com - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
pywikipedia-zzgallery.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
michiel_copylabel (wikipedia:en) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
target (wikipedia:en; User:Sgabrovski) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.8.final.0 - php=zend
 application/json
iw2data/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
PyApp1-run_movepages.2.py/r11068 Pywikipediabot/1.0 - php=zend
 application/json
5453030 (wikipedia:en; User:RussBot) Pywikibot/2.0b2 (g3) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - php=hhvm
 application/json
pywikipedia-lit.py/rg11101 Pywikipediabot/1.0 - php=zend
 application/json
botjagwar-getpron.py/r11215 Pywikipediabot/1.0 - php=zend
 application/json
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
gsa-crawler (Enterprise; T3-GJ67B6PT9YWQG; mail address ) - php=zend
 text/..
mrmetadata (wikipedia:fr) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
5879703 (wikipedia:en; User:RussBot) Pywikibot/2.0b2 (g3) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Screaming Frog SEO Spider/2,55 - php=zend
 text/..
WikiBot/0.1 - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
welcome-c (wikipedia:fr; User:Loveless) Pywikibot/2.0b2 (g4483) httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; fr-fr; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 fr-FR, en-US -
 image/..
renametemp (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
pywikipedia-git-maghalesaz_new.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
touch (wikipedia:ar; User:ZkBot) Pywikibot/2.0b2 (g4556) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
 application/xml
pywikipedia-commonscat.py/rg10961 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
mrmetadata (wikipedia:he) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
PyApp1-run_movepages.1.py/r11068 Pywikipediabot/1.0 - php=zend
 application/json
YOUBot (Youniversal Media Bot) - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
BotReversor-BotReversor.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-lit.py/rg11089 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
pywikilab-llista_mils.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
redirect/-1 (unknown) Pywikibot/2.0 (User:Rahl43bot) - php=zend
 application/json
mrmetadata (wikipedia:ja) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
JackBot-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-reportBLP.py/r10494 Pywikipediabot/1.0 - php=zend
 application/json
spider - php=hhvm
 text/..
 application/json
 application/vnd.php.serialized
run_replace.7r.03/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
Omnindexbot/1.0 en, fr,* php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
insertDistriWD (wikidata:wikidata; User:Symac_bot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
replace (wikisource:ru; User:HinoteBot) Pywikibot/2.0b1 (g4018) httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
pywikipedia-git-featured_new.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
 application/xml
WikidataBot framework - php=hhvm
 application/json
PythonWikipediaBot/1.0 - php=hhvm
 application/json
 text/..
NutchSpider/Nutch-1.10-SNAPSHOT (Nutch crawler used for research purpose at Curtin university.) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
pywikibot-compat-redirect.py/rg10993 Pywikipediabot/1.0 - php=zend
 application/json
delete_template (wikipedia:ru; User:WinterheartBot) Pywikibot/2.0b2 (g4435) httplib2/0.9 Python/2.7.7.final.0 - php=zend
 application/json
PHPCrawl - php=zend
 text/..
 application/rsd+xml
milog_bot/1.0 ( mail address ) - php=hhvm
 text/..
RSS Bot/1.9 (Mac OS X Version 10.10 (Build 14A389)) en;q=1 php=zend
 application/xml
 -
run_replace.7r.02/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
cd-delinker.py/r11775 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-wikifyer.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
Screaming Frog SEO Spider/2,21 - php=zend
 text/..
rapid-miner-crawler - php=zend
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - php=hhvm
 application/json
AnomieBOT/1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - php=zend
 application/json
mrmetadata (wikipedia:uk) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - php=hhvm
 application/json
 text/..
node.js - nodemw - mail address - Maintainer: Rainer Rillke - mail address - [[:commons:User:Rillke]] - php=zend
 application/json
pywikibot-1newar25.py/rg11106 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-zzaddnavebox.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-zzget_item_2.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
nutch spider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
templatedatafetcher (wikipedia:fr) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.8.final.0 - php=zend
 application/json
 text/..
DotNetWikiBot/3.11 (Microsoft Windows NT 6.2.9200.0; ) - php=hhvm
 text/..
Mozilla/5.0 (Apibot Browser) - php=zend
 application/json
 application/vnd.php.serialized
interwiki (wikinews:fr) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
MyCuteBot/0.1 - php=hhvm
 text/..
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - php=zend
 text/..
MoodleBot/1.0 - php=zend
 application/xml
 text/..
 application/vnd.php.serialized
pywikipedia-git-radesaz.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
HTMLParser/2.0 - php=hhvm
 text/..
getDiseaseOntology/rg3599 Pywikibot/2.0 - php=zend
 application/json
PythonWikipediaBot/1.0 -
 application/json
 text/..
touch (wiktionary:cs; User:JAnDbot) Pywikibot/2.0b2 (s5607) httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
compat-label_bot.py/rg11051 Pywikipediabot/1.0 - php=hhvm
 application/json
AnomieBOT/1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - php=hhvm
 application/json
SEOSTMSpider/Nutch-1.7 (SEOSTMSpider) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" - -
 image/..
 text/..
MediaWiki::Perl/User:Wdwdbot - php=zend
 application/json
bar (wikipedia:it; User:IncolaBot) Pywikibot/2.0b2 (g3) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Omnindex.bot/1.0 en,ro,fr php=zend
 text/..
ko_missing (wikipedia:en; User:Chobot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
SPIupdater-globalfunc.py/r10417 Pywikipediabot/1.0 - php=hhvm
 application/json
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - php=zend
 text/..
 application/xml
 application/json
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - php=hhvm
 application/json
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - php=zend
 application/xml
 text/..
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us php=zend
 text/..
compat-pwb.py/rg11092 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
 application/xml
pywikipedia-git-sectionBotit.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - php=hhvm
 text/..
 application/xml
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
getActeursQ (wikipedia:fr) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
CaBot - php=zend
 text/..
com.apple.Safari.WebFeedParser/600.1.4 CFNetwork/711.1.12 Darwin/14.0.0 en-us php=zend
 application/xml
 text/..
pywikipedia-newar.py/rg11089 Pywikipediabot/1.0 - php=zend
 application/json
compat-redirect.py/rs11076 Pywikipediabot/1.0 - php=hhvm
 application/json
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - php=hhvm
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* php=hhvm
 text/..
 image/..
 application/json
ITeco bot, v0.0, mail address - php=zend
 text/..
synspider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
node-crawler/0.2.5 - php=hhvm
 text/..
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b2 (3) httplib2/0.9 Python/2.7.8.final.0 - php=hhvm
 application/json
AnomieBOT/1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - php=hhvm
 application/json
pywikibot-1newar25.py/rg11101 Pywikipediabot/1.0 - php=hhvm
 application/json
compat-checkimages.py/rs11076 Pywikipediabot/1.0 - php=hhvm
 application/json
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b2 (4429) httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
vm-auto-erl-neu (wikipedia:de; User:Xqbot) Pywikibot/2.0b2 (s5459) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
interwiki (wikinews:pt) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US php=zend
 text/..
 -
 image/..
 application/json
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - php=hhvm
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US php=zend
 text/..
 -
 image/..
 application/json
Projet Sport-Projetsportliste2.py/r11052 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikibot-compat-sichteroptin.py/r10883 Pywikipediabot/1.0 - php=zend
 text/..
 application/json
Owlin Bot v. 3.0 - php=hhvm
 text/..
 application/xml
 -
getpages (wikipedia:de; User:Wolfgang_Liebig) Pywikibot/2.0b2 (-1 (unknown)) httplib2/0.8 Python/2.7.8.final.0 - php=zend
 application/json
pywikipedia-interwiki.py/rg42 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
mrmetadata (wikipedia:fi) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
Wikibot/2.1.1 CFNetwork/672.1.15 Darwin/14.0.0 en-us -
 image/..
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b2 (4437) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
python (wikipedia:cs; User:Frettiebot) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
Mozilla/4.5(compatible;Omnindexbot/1.0) en, * php=zend
 text/..
 application/xml
Arachnida Web Crawler - php=zend
 text/..
getCastingFromWp (wikipedia:fr) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - php=hhvm
 application/json
welcome (wikipedia:sr; User:MilicevicBot) Pywikibot/2.0b2 (4428) httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
redirect (wikipedia:de; User:Xqbot) Pywikibot/2.0b2 (g4606) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
moogolbot/Moogol-1.0 (moogolbot) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
addbot (wikidata:wikidata; User:YiFeiBot) Pywikibot/2.0b2 (g4556) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
 text/..
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
Ziyou Crawler/Nutch-1.9 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - php=zend
 application/json
 text/..
MoodleBot/1.0 -
 text/..
 application/xml
 application/vnd.php.serialized
pywikipedia-git-new_iw14.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
redirect (wikipedia:en; User:Xqbot) Pywikibot/2.0b2 (s5587) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - php=zend
 application/json
Baiduspider - php=zend
 image/..
 text/..
compat-basictemplatedisamblist.py/rg11090 Pywikipediabot/1.0 - php=hhvm
 application/json
wiktionaryInterwiki (wiktionary:mg; User:Bot-Jagwar) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
compat-basicdisambrefslist.py/rg11090 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
ContextAd Bot 1.0 - php=hhvm
 text/..
GermCrawler - php=hhvm
 application/json
 application/xml
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US php=zend
 -
 text/..
 image/..
 application/json
RSS Bot/1.9 (Mac OS X Version 10.10.1 (Build 14B25)) en;q=1 php=zend
 application/xml
 -
pwb/rg3045 Pywikipediabot/2.0 - php=zend
 application/json
pywikipedia-zzporb-main.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
pwb/rg3012 Pywikipediabot/2.0 - php=zend
 application/json
pywikipedia-newar.py/rg11037 Pywikipediabot/1.0 - php=zend
 application/json
EarwigBot/0.2.devgit-07bbf240 (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - php=zend
 application/json
iw_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
mrmetadata (wikipedia:hu) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
compat-basicsbz.py/rg11090 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
GoogleBot-Image/1.0 - php=hhvm
 text/..
 application/json
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b2 (4429) httplib2/0.9 Python/2.7.8.final.0 - php=zend
 application/json
update-task-categories (wikipedia:en; User:SuggestBot) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.1.final.0 - php=zend
 application/json
addbot (wikipedia:en; User:YiFeiBot) Pywikibot/2.0b2 (g4556) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
IssueCrawler - php=zend
 text/..
Wikibot/2.1.1 CFNetwork/711.1.12 Darwin/14.0.0 en-us -
 image/..
 text/..
addbot (commons:commons; User:YiFeiBot) Pywikibot/2.0b2 (g4605) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US php=zend
 -
 text/..
 image/..
 application/json
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - php=hhvm
 text/..
mrmetadata (wikipedia:zh) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
PerlWikiBot/5.006002 (https://ca.wikipedia.org/wiki/Usuari:Langtoolbot; User:Jaumeortola) - php=zend
 application/json
pywikipedia-git-sectionBoten.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
analysiq_spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
images (wikipedia:es) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
pywikipedia-bot_rss.py/rg14395 Pywikipediabot/1.0 - php=zend
 application/json
AnomieBOT/1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
mrmetadata (wikipedia:pt) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
redirect (wikipedia:en; User:Xqbot) Pywikibot/2.0b2 (s5640) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
insertFilmsIntoWD (wikidata:wikidata; User:Symac_bot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Update_cats_and_iwiki (commons:commons) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US php=zend
 text/..
 -
 image/..
 application/json
YOUBot (Youniversal Media Bot) - -
 image/..
GoogleBot - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
compat-welcome.py/rg11087 Pywikipediabot/1.0 - php=zend
 application/json
run (wikipedia:fr) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - php=zend
 application/xml
 text/..
articles (wikipedia:fr; User:Peter17-Bot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
vocalize (wiktionary:en; User:WingerBot) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.8.final.0 - php=zend
 application/json
29450.14total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Dec 8, 2014 14:26
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers