Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Dec 2014 - 31 Dec 2014
000 ⇒ k
 

 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 145,075,840 page requests (mime type text/html only!) per day are considered crawler requests, out of 664,640,000 external requests, which is 21.8%

Page requests for crawlers that specify a url in the agent string
Count
x 1000
Secondary domain
(~site) name
URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm;https=1
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1;php=hhvm
 www.bing.com/bingbot.htm-Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - php=hhvm;https=1
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - php=hhvm
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - https=1;php=hhvm
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1;php=zend
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - php=hhvm
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm;https=1
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm;https=1
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm;https=1
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm;https=1
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~guidesites-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6001-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile8-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~cgiproxy3-hrd) - php=hhvm
 www.bing.com/bingbot.htm-Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - php=zend;https=1
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; url) - https=1;php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~usethisproxy-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~lookingforwe-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy4-hrd) - php=hhvm
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~payonlyproxy-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf306-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible;bingbot/2.0;url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~overviewa-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk9-hrd) - php=hhvm
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmimage/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - php=hhvm
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~goodersearch-hrd) - php=hhvm
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - php=hhvm
 code.google.com/p/crawler4j/text/..crawler4j (url) - php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - php=hhvm;https=1
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=hhvm
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=hhvm
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=hhvm
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~jnhanproxy) - php=hhvm
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - php=hhvm
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 code.google.com/p/crawler4j/text/..crawler4j (url) - php=zend
 code.google.com/p/crawler4j/text/..crawler4j (url) - php=hhvm;https=1
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl0) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jayjudah1) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl2) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq2) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp5) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp2) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq7) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadl3) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadl) - php=hhvm
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadl0) - php=hhvm
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - php=hhvm
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools3) - php=hhvm
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 code.google.com/p/crawler4j/image/..crawler4j (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools5) - php=hhvm
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq) - php=hhvm
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp2) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vibhoreindia) - php=hhvm
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: jayjudah1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - php=hhvm
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - php=hhvm
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~harunakaze) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~223-p1) zh-CN,zh;q=0.8,en;q=0.6 php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~vibhoreindia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - php=hhvm
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - php=hhvm
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~449-lsw3) zh-CN,zh;q=0.8,en;q=0.6 php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~argentina-country) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - php=hhvm
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=hhvm
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - php=hhvm
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: chronickg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~e-ddos) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - php=hhvm
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - php=hhvm
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn03) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~223-p7) zh-CN,zh;q=0.8,en;q=0.6 php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: go-online-now) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - php=hhvm
 code.google.com/p/gwtwiki/text/..JavaWikipediaAPI/3.1-SNAPSHOT url - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ungoodthinkful) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - php=hhvm
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; spreadsheets; url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: unblock4myspace) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~jnhanproxy) - php=zend
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - php=hhvm
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~safebrowsing88) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chronickg) - php=hhvm
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - php=hhvm
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: goeverysite) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: its-a-tr) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bobolaw1) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - php=hhvm
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~223-p4) zh-CN,zh;q=0.8,en;q=0.6 php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~jnhanproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lil05warrior) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - php=hhvm
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: luigi-tiburzio) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: carlos-web-proxy) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps002) - php=hhvm
 code.google.com/p/ldspider/wiki/Robotstext/..ldspider (url) - php=hhvm
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - php=hhvm
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - php=hhvm
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - php=hhvm;https=1
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 php=hhvm
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - php=hhvm
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=hhvm
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - php=hhvm;https=1
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 php=hhvm;https=1
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1;php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spi_der.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en,* php=hhvm
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US php=hhvm;https=1
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - php=hhvm
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en,* php=hhvm
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - php=hhvm;https=1
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=hhvm
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US https=1;php=hhvm
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) zh-cn,zh-tw php=hhvm
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - php=zend
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1;php=zend
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.5b5 - php=hhvm
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en,* php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US,* php=hhvm
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - php=hhvm
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - php=hhvm;https=1
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - php=zend
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - php=hhvm
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* php=hhvm
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - php=hhvm;https=1
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1;php=hhvm
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - php=zend
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X; ja-jp) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Y!J-ASR/1.0 fetcher (Y!J-AGENT; url)) en-US,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* php=zend
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Mozilla/5.0 (compatible; Y!J-ASR/1.0 fetcher (Y!J-AGENT; url)) - php=hhvm
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X; ja-jp) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Y!J-ASR/1.0 fetcher (Y!J-AGENT; url)) en,* -
 help.yahoo.com/help/us/ysearch/slurpvideo/webmMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm;https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=hhvm
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=hhvm
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=hhvm
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - php=hhvm;https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-us,zh;q=0.9 https=1;php=hhvm
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend;https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - https=1;php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=hhvm
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - https=1;php=hhvm
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 php=hhvm
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend;https=1
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmltext/..54.208.57.30 Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm;https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8,bn;q=0.6 php=zend
 www.google.com/bot.htmltext/..54.236.100.183 Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm;https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=zend
 www.google.com/bot.htmlimage/..GoogleBot/2.1 (url) en,* -
 www.google.com/bot.htmltext/..54.208.37.246 Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm;https=1
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..54.208.37.89 Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=hhvm;https=1
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend;https=1
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8,bn;q=0.6 php=hhvm
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-us,zh;q=0.9 https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=hhvm;https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US php=zend
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend;https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 php=zend
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ru,en-us;q=0.7,en;q=0.7 php=hhvm
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg php=zend
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - php=hhvm
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - php=zend
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=hhvm
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..127.0.0.5 Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=hhvm
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..127.0.0.5 Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) tr, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..127.0.0.5 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..127.0.0.5 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - php=hhvm
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1;php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - php=hhvm;https=1
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..127.0.0.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=hhvm;https=1
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - php=hhvm
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - php=hhvm
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=hhvm;https=1
 yandex.com/botstext/..127.0.0.5 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4
 yandex.com/botstext/..127.0.0.5 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4 php=hhvm
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 php=hhvm
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=hhvm
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=hhvm;https=1
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=hhvm
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - php=hhvm
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - https=1;php=hhvm
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=zend
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - php=hhvm
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - php=zend
cibra
 cibra.de/text/..CiBra Data Collector (url) - php=hhvm
 cibra.de/text/..CiBra Data Collector (url) - php=zend
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - php=hhvm
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - php=hhvm
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - php=zend
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - php=zend
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
msn
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - php=hhvm
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - php=hhvm
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - php=hhvm
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - php=hhvm
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - php=zend
naver
 help.naver.com/robots/image/..Mozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/application/jsonMozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=hhvm
 help.naver.com/robots/application/jsonMozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
 help.naver.com/robots/text/..Mozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=hhvm
 help.naver.com/robots/text/..Mozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 https=1;php=hhvm
 help.naver.com/robots/application/oggMozilla/5.0 (compatible; Yeti/1.1; url) en-US,* -
 help.naver.com/robots/application/oggMozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en php=hhvm
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en php=hhvm
 the.echonest.com/reader/application/xmlnestReader/0.4 (discovery; url; reader at echonest.com) en php=hhvm
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/application/xmlnestReader/0.4 (discovery; url; reader at echonest.com) en php=zend
wikiwand
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=hhvm
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=hhvm
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.wikiwand.comimage/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=hhvm
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=zend
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - php=hhvm
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=hhvm
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=hhvm;https=1
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - php=hhvm
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - php=hhvm;https=1
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - php=hhvm
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - php=zend
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/MusikAnimal url en-US,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* php=hhvm;https=1
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stickee url en-AU,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stickee url en-AU,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/MusikAnimal url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/haminoon url en-NZ,* php=hhvm;https=1
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.26.0 url - php=hhvm
 ko.wikipedia.orgapplication/jsonurl - php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/HeicoH url de-DE,en,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Vitor_Mazuco url pt-BR,en,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/wtmitchell url en-US,* php=hhvm;https=1
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Becky_Sayles url en-US,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Isha url es-AR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Winner_42 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/KylieTastic url en-GB,* php=hhvm;https=1
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Vitor_Mazuco url pt-BR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/HelloThereMinions url en-CA,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/HeicoH url de-DE,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Vitor_Mazuco url pt-BR,en,* php=hhvm;https=1
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/wtmitchell url en-US,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Manuel_Anastácio url pt-PT,en,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/I_dream_of_horses url en-US,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Manslay url ru-RU,en,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Spuk968 url de-DE,en,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Serols url de-DE,en,* php=hhvm;https=1
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Winner_42 url en-US,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Egis57 url es-AR,en,* php=hhvm
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Rrburke url en-CA,* php=zend
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - php=hhvm
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - php=hhvm
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot/2.X (urlGoogleBot.com/bot.html) - php=hhvm
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - php=hhvm
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) -
 www.text/..Mozilla/5.0 (compatible; GoogleBot/2.1 urlGoogleBot.com/bot.html) - php=hhvm
quickiwiki
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - php=hhvm
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn php=hhvm
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - php=hhvm
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn php=zend
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - php=hhvm
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - php=zend
edu
 pjwstk.edu.pltext/..Wiki Controversial/0.1; url ; mail address -
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - php=hhvm
commoncrawl
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
yodao
 www.yodao.com/help/webmaster/spider/application/json(compatible; YodaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 php=hhvm
 www.yodao.com/help/webmaster/spider/application/json(compatible; YodaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 php=zend
localhost
 localhostapplication/jsonLagotto 3.6.3 - url - php=hhvm
 localhostapplication/jsonLagotto 3.6.3 - url - php=zend
 localhost/bot.phptext/..Mozilla/5.0 (compatible; CrawlRobot; url) - php=hhvm
 localhost/wordpresstext/..WordPress/4.0.1; url - php=hhvm
 localhost/Tinnitus/wordpresstext/..WordPress/4.0.1; url - php=hhvm
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 php=hhvm
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 php=hhvm;https=1
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 php=zend
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - php=hhvm
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 https=1;php=hhvm
 help.coccoc.com/application/jsonMozilla/5.0 (compatible; coccoc/1.0; url) - php=hhvm
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: go2-uk2) en-us;q=0.7,en;q=0.3 php=hhvm
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: go2-vn) en-us;q=0.7,en;q=0.3 php=hhvm
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs php=hhvm
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) cs php=hhvm
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs php=hhvm
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 php=hhvm
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs php=zend
 fulltext.sblog.cz/image/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) cs php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs php=zend
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; url) en,* -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) - -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 php=zend
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - php=hhvm
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - php=zend
 pear.php.net/package/http_request2text/..HTTP_Request2/2.0.0 (url) PHP/5.5.18 ja php=hhvm
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - php=hhvm
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - php=hhvm
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.29 - php=hhvm
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.19 - -
plos
 alm.plos.orgapplication/jsonLagotto 3.9.6 - url - php=hhvm
 alm.plos.orgapplication/jsonLagotto 3.11 - url - php=hhvm
 sfo-stage-alm01.int.plos.orgapplication/jsonLagotto 3.9.1 - url - php=zend
 sfo-stage-alm01.int.plos.orgapplication/jsonLagotto 3.9.6 - url - php=hhvm
 alm.plos.orgapplication/jsonLagotto 3.10 - url - php=hhvm
 sfo-stage-alm01.int.plos.orgapplication/jsonLagotto 3.9.1 - url - php=hhvm
 alm.plos.orgapplication/jsonLagotto 3.9.6 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.9.4 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.9.4 - url - php=hhvm
 alm.plos.orgapplication/jsonLagotto 3.9.3 - url - php=zend
 dlm.plos.orgapplication/jsonLagotto 3.9.3 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.9.8 - url - php=hhvm
 alm.plos.orgapplication/jsonLagotto 3.9.3 - url - php=hhvm
 dlm.plos.orgapplication/jsonLagotto 3.9.3 - url - php=hhvm
 alm.plos.orgapplication/jsonLagotto 3.9.5 - url - php=zend
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - php=hhvm
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - php=zend
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - php=hhvm
 tools.wmflabs.org/geohacktext/..Geohack (url) - php=zend
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - php=hhvm
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - php=hhvm;https=1
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - php=zend
 tools.wmflabs.org/kmlexporttext/..url libwww-perl/6.03 - php=hhvm
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - php=hhvm
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - php=hhvm
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - php=zend
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - php=zend
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - php=hhvm
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; Please Name Your robot; url) - php=hhvm
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - php=zend
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) -
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - php=hhvm;https=1
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; Please Name Your robot; url) - php=zend
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.18-398.el5; java 1.6.0_33; Asia/ja) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.18-398.el5; java 1.6.0_33; Asia/ja) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.17.0-031700-generic; java 1.7.0_65; Asia/ja) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.0-25-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.6-2-desktop; java 1.8.0_40; Europe/de) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_51; UTC/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.0-25-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 8 6.2; java 1.7.0_71; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.5.0-44-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p15; java 1.7.0_71; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-43-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.7.10-1.40-desktop; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.17.0-031700-generic; java 1.7.0_65; Asia/ja) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.16.5-gentoo; java 1.7.0_71; GMT/ja) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Linux 3.17.4; java 1.7.0_71; Europe/es) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.6-2-desktop; java 1.8.0_40; Europe/nl) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_17; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmlimage/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_51; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.17.4; java 1.8.0_25; Europe/ru) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-504.el6.x86_64; java 1.7.0_71; America/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_65; Europe/pl) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; i386 Linux 3.2.0-4-686-pae; java 1.7.0_65; Europe/fi) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 8 6.2; java 1.8.0_25; Europe/de) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (freeworld/global; PowerPC OS/400 V7R1M0; java 1.6.0; UTC/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/fr) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86_64 Mac OS X 10.7.5; java 1.6.0_65; Australia/en) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; America/en) url en-us,en;q=0.5 php=hhvm;https=1
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.7.10-1.40-desktop; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 php=hhvm
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=hhvm;https=1
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - php=hhvm
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - php=hhvm
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - php=zend
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - php=zend
scrapy
 scrapy.orgtext/..Scrapy/0.25.0-232-ged84231 (url) en php=hhvm
 scrapy.orgtext/..Scrapy/0.24.4 (url) en php=hhvm
 scrapy.orgtext/..Scrapy/0.25.0-232-ged84231 (url) en php=hhvm;https=1
 scrapy.orgtext/..Scrapy/0.22.2 (url) en php=hhvm
 scrapy.orgtext/..Scrapy/0.25.1 (url) en php=hhvm;https=1
 scrapy.orgimage/..Scrapy/0.24.4 (url) en -
 scrapy.orgtext/..Scrapy/0.25.0-232-ged84231 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.24.4 (url) en https=1;php=hhvm
 scrapy.orgtext/..Scrapy/0.24.4 (url) en php=hhvm;https=1
etao
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - php=hhvm
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - php=zend
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - php=hhvm;https=1
tnode
 mediawiki.p.tnode.com/application/jsonWikiMedia Meteor DDP stream (url, mail address ) - php=hhvm
 example.p.tnode.com/application/jsonWikiMedia Meteor DDP stream (url, mail address ) - php=hhvm
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - php=hhvm
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en php=hhvm
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) ru php=hhvm
 www.majestic12.co.uk/bot.php?image/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en php=zend
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - php=zend
 www.majestic12.co.uk/bot.php?application/jsonMozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - php=hhvm
pagefreezer
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 php=hhvm
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 php=zend
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 php=hhvm;https=1
xbmc
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.3;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.0.36 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140604-84725b0 (Linux; Android; 3.10.33 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.10.33 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (iOS; 13.0.0, Version 6.1.4 (Build 10B809); url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140821-c5ae8b3 (Linux; Android; 3.10.33 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140831-a1505b1 (Linux; Android; 3.10.33 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.4.0-perf-g21e3850 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.0 Git:20140505-a394e15 (Linux; Android; 3.0.36 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140607-49d7d62 (Linux; Android; 3.10.33 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:0f3db05 (Linux; OpenELEC (official) - Version: 4.2.1; 3.16.3 x86_64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Mac OS X; 14.0.0 x86_64, Version 10.10.1 (Build 14B25); url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.4.0-perf-gf960815 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.3-MINIX(0) Git:20140423-b8bc5a0 (Linux; Android; 3.10.33 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.3-SPMC(2) Git:20140423-b8bc5a0 (Linux; Android; 3.0.36 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.0 Git:20140505-f42f5b5 (Linux; Android; 3.0.50 armv7l; url) - -
meteor
 wikimedia.meteor.com/application/jsonWikiMedia Meteor DDP stream (url, mail address ) - php=hhvm
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - php=hhvm
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - php=hhvm
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - php=zend
zeerch
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - php=hhvm
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - php=hhvm
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - php=hhvm;https=1
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - php=zend
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - php=zend
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - https=1;php=hhvm
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - php=hhvm;https=1
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - php=hhvm
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - php=hhvm;https=1
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - php=zend
flipboard
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 php=hhvm
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 php=zend
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 php=hhvm;https=1
wordpress
 dshenai.wordpress.comtext/..WordPress/4.1-beta2-30771; url - php=hhvm
 goldendragon1971.wordpress.comtext/..WordPress/4.1-beta2-30771; url - php=hhvm
 raymondpronk.wordpress.comtext/..WordPress/4.1-beta2-30640; url - php=hhvm
 zeitgeist77.wordpress.comtext/..WordPress/4.1-beta2-30797; url - php=hhvm
 vip2099.wordpress.comtext/..WordPress/4.1-beta2-30771; url - php=hhvm
sputnik
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 php=hhvm
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 php=zend
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=hhvm
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=hhvm
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=zend
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=hhvm;https=1
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=hhvm;https=1
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=zend
github
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/1.9.3p551 (url) en-us,en;q=0.5 php=hhvm
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 php=hhvm
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/1.9.3p551 (url) en-us,en;q=0.5 php=zend
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 php=zend
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - php=hhvm
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - php=hhvm
 wiki.github.com/bixo/bixo/bixocrawlertext/..Mozilla/5.0 (compatible; pub-crawler; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - php=hhvm
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 php=hhvm
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - php=hhvm
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - php=zend
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de-DE, de;q=0.9, dsb-DE;q=0.3, hsb-DE;q=0.2 php=hhvm
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de-DE, de;q=0.9, dsb-DE;q=0.3, hsb-DE;q=0.2 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) fr-FR, fr;q=0.9, br-FR;q=0.7, gsw-FR;q=0.5, co-FR;q=0.3 php=hhvm
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - php=hhvm
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - php=zend
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - php=hhvm
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - php=zend
 www.monitis.comimage/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - php=hhvm
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.2.2 (url; mail address en-us php=hhvm
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-us php=hhvm
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-us php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.2.2 (url; mail address en-us php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.2.2 (url; mail address en-gb php=hhvm
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.2.2 (url; mail address en-us
example
 www.example.com/text/..wikidrain/1.0 (url) - php=hhvm
 www.example.com/text/..wikidrain/1.0 (url) - php=zend
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - php=hhvm
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - php=zend
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - php=hhvm
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - php=hhvm
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - php=hhvm;https=1
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - php=hhvm;https=1
 archive.orgimage/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
 archive.orgtext/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - php=hhvm
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 url) - php=hhvm
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - php=hhvm
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - php=zend
 www.opensiteexplorer.org/dotbotapplication/jsonMozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - php=hhvm
wesee
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PageBot (url) - php=hhvm
drupal
 drupal.org/application/jsonDrupal (url) - php=hhvm
 drupal.org/text/..Drupal (url) - php=hhvm
 drupal.org/application/jsonDrupal (url) - php=zend
 drupal.org/text/..Drupal (url) - php=zend
 drupal.org/text/..Drupal (url) -
 drupal.org/image/..Drupal (url) - -
tineye
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - php=hhvm
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - php=zend
riddler
 riddler.io/abouttext/..Riddler (url) - php=hhvm
 riddler.io/abouttext/..Riddler (url) - php=zend
test
 test.co.kr/text/..xxxxxxxx/Nutch-2.2.1 (url) - php=hhvm
 test.co.kr/text/..xxxxxxxx/Nutch-2.2.1 (url) - php=zend
recuweb
 recuweb.comapplication/jsonurl en-us,en;q=0.5 php=hhvm;https=1
 recuweb.comapplication/jsonurl en-us,en;q=0.5 php=hhvm
 recuweb.comapplication/jsonurl en-us,en;q=0.5 php=zend
uni-leipzig
 corpora.informatik.uni-leipzig.de/crawler_faq.htmltext/..Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV url) - php=hhvm
 corpora.informatik.uni-leipzig.de/crawler_faq.htmltext/..Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV url) - php=hhvm;https=1
 corpora.informatik.uni-leipzig.de/crawler_faq.htmltext/..Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV url) - https=1;php=hhvm
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * php=hhvm
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - php=hhvm
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - php=zend
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - php=hhvm;https=1
 www.feedly.com/fetcher.htmltext/..Feedly/1.0 (url; like FeedFetcher-Google) - php=hhvm
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - php=hhvm
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
 duckduckgo.comtext/..Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; url) en-US,en;q=0.8,zh;q=0.6,es;q=0.4 php=hhvm
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 php=hhvm
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 php=zend
bigdatacorp
 ecommerce.bigdatacorp.com.br/faq.aspxtext/..Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; url) AppleWebKit/535.1 KHTML Chrome/13.0.782.107 Safari/535.1 pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4 php=hhvm
 ecommerce.bigdatacorp.com.br/faq.aspxtext/..Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; url) AppleWebKit/535.1 KHTML Chrome/13.0.782.107 Safari/535.1 pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=hhvm
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=hhvm;https=1
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=zend
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - php=hhvm
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - php=zend
netvibes
 www.netvibes.comtext/..Netvibes (url) - php=hhvm
 www.netvibes.comapplication/xmlNetvibes (url) - php=hhvm
 www.netvibes.comapplication/jsonNetvibes (url) en-us php=hhvm
 www.netvibes.comtext/..Netvibes (url) - php=zend
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=hhvm
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=zend
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - php=hhvm
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - php=zend
mysite
 www.mysite.com/application/vnd.php.serializedMyBot/1.0 (url) - php=hhvm
 www.mysite.com/application/jsonMyBot/1.0 (url) - php=hhvm
 www.mysite.com/text/..MyBot/1.0 (url) - php=hhvm
 www.mysite.com/application/jsonMyBot/1.0 (url) - php=zend
 www.mysite.com/application/vnd.php.serializedMyBot/1.0 (url) - php=zend
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - php=hhvm
pchome
 www.pchome.com.tw/pchomebot.htmtext/..Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 php=hhvm
v--u
 v--u.com/cookingtext/..WordPress/4.1; url - php=hhvm
 v--u.com/lotterytext/..WordPress/4.1; url - php=hhvm
 v--u.com/lawyerstext/..WordPress/4.1; url - php=hhvm
 v--u.com/acnetext/..WordPress/4.1; url - php=hhvm
 www.v--u.com/hemorrhoidstext/..WordPress/4.0.1; url - php=hhvm
 v--u.com/back-paintext/..WordPress/4.0.1; url - php=hhvm
 v--u.com/depressiontext/..WordPress/4.0.1; url - php=hhvm
 v--u.com/nutritiontext/..WordPress/4.0.1; url - php=hhvm
 v--u.com/insomniatext/..WordPress/4.0.1; url - php=hhvm
 v--u.com/juicingtext/..WordPress/4.0.1; url - php=hhvm
 v--u.com/lotterytext/..WordPress/4.0.1; url - php=hhvm
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - php=hhvm
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - php=zend
okian
 www.okian.ro/text/..MyBot/1.0 (url) - php=hhvm
 www.okian.ro/application/jsonMyBot/1.0 (url) - php=hhvm
 www.okian.ro/text/..MyBot/1.0 (url) - php=zend
 www.okian.ro/application/jsonMyBot/1.0 (url) - php=zend
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - php=hhvm
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - php=zend
grouphigh
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - php=hhvm
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - php=zend
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - php=hhvm;https=1
 www.grouphigh.com/application/xmlMozilla/5.0 (compatible; GroupHigh/1.0; url) - php=hhvm
datasift
 datasift.com/bot.htmltext/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - php=hhvm
 datasift.com/bot.htmltext/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - php=zend
 datasift.com/bot.htmlimage/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - -
 datasift.com/bot.htmltext/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - php=hhvm;https=1
yapq
 yapq.comapplication/jsonYapq mail address url - php=hhvm
 yapq.comapplication/jsonYapq mail address url - php=zend
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - php=hhvm
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - php=hhvm
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - php=zend
netarkivet
 netarkivet.dk/webcrawlerimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawlertext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=hhvm
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=hhvm
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
draigweb
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - php=hhvm
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - php=zend
amzn
 amzn.to/1vsZADitext/..Amazon Route 53 Health Check Service; ref:75c68de8-ca59-474b-a8a5-1bcc0ec34b5d; report url - php=hhvm;https=1
 amzn.to/1vsZADitext/..Amazon Route 53 Health Check Service; ref:75c68de8-ca59-474b-a8a5-1bcc0ec34b5d; report url - php=hhvm
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - php=hhvm
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - php=zend
web
 www.web.nl/text/..web.nl spider/1.7; url - php=hhvm
noblis
 www.noblis.orgtext/..beholder (url) en php=hhvm
 www.noblis.orgimage/..ISSbot (url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * php=hhvm
wikidict
 www.wikidict.detext/..url - php=hhvm
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - php=hhvm
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 php=hhvm
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
pinterest
 www.pinterest.com/text/..Pinterest/0.2 (url) - php=hhvm
 www.pinterest.com/image/..Pinterest/0.2 (url) - -
 pinterest.com/image/..Pinterest/0.1 url - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - php=hhvm
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.4.0; url ) en-us,en;q=0.5 php=hhvm
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.4.2; url ) en-us,en;q=0.5 php=hhvm
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.4.1; url ) en-us,en;q=0.5 php=hhvm
nipponpaint
 nipponpaint.com.vntext/..WordPress/3.5.1; url - php=hhvm
 nipponpaint.com.vntext/..WordPress/3.5.1; url - php=zend
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) -
yahoo-help
 www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/text/..Y!J-ASR/0.1 crawler (url) ja;q=1.0,en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
bit
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) de,en php=hhvm
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - php=hhvm
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - php=hhvm
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - php=hhvm
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - php=hhvm
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - php=hhvm
 www.alexa.com/help/webmasterstext/..Mozilla/5.0 (compatible; alexa site audit/1.0; url; mail address ) - php=hhvm
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - php=hhvm
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - php=hhvm
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - php=zend
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) it php=hhvm
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) php=hhvm
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en php=hhvm
 bsurprised.com/text/..BSurprised WikiBox 0.1 (url) en php=hhvm
Anonymouse
 Anonymouse.org/text/..url (Unix) - php=hhvm
 Anonymouse.org/image/..url (Unix) - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - php=hhvm
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - php=zend
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - php=hhvm
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - php=hhvm
crazywebcrawler
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.1, url - php=hhvm
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.2, url - php=hhvm
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.1, url - php=zend
amdaris
 www.amdaris.comapplication/jsonAmdarisVirtualTour/1.0 (url; mail address ) - php=zend
 www.amdaris.comapplication/jsonAmdarisVirtualTour/1.0 (url; mail address ) - php=hhvm
leiki
 www.leiki.comtext/..Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; url) - php=hhvm
 www.leiki.comtext/..Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; url) - php=zend
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - php=hhvm
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - php=zend
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - php=hhvm
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
website
 websitetext/..TheNameOfMyBot/1.0 (url, mail address ) LinqToWiki - php=hhvm
 websitetext/..TheNameOfMyBot/1.0 (url, mail address ) LinqToWiki -
domeinwinkel
 www.domeinwinkel.nl/text/..Domeinwinkel CatchCrawler/1.0 (url) -
newworldmodel
 newworldmodel.com/auto-insurancetext/..WordPress/4.1; url - php=hhvm
 newworldmodel.com/couponstext/..WordPress/4.1; url - php=hhvm
 www.newworldmodel.com/hemorrhoidstext/..WordPress/4.0.1; url - php=hhvm
 newworldmodel.com/nutritiontext/..WordPress/4.0.1; url - php=hhvm
gorgor
 gorgor.irtext/..hivaBot/hivaBot-1.0 (Iranian Search Engine bot; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 gorgor.irapplication/jsonhivaBot/hivaBot-1.0 (Iranian Search Engine bot; url; mail address ) - php=hhvm
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) -
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - php=hhvm
archiwwwe
 www.archiwwwe.com/bot.htmtext/..Mozilla/5.0 (compatible; Archiwwwebot/1.0; url) en, * php=hhvm;https=1
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - php=hhvm
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - php=hhvm;https=1
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
friedemann-vogel
 friedemann-vogel.de/forschung/konfliktlinguistik-20application/jsonTalksCrawler/2.0 (url) - php=hhvm
 friedemann-vogel.de/forschung/konfliktlinguistik-20application/jsonTalksCrawler/2.0 (url) - php=zend
esophoria
 www.esophoria.orgtext/..WordPress/3.4.2; url - php=hhvm
themself
 www.themself.orgtext/..WordPress/4.1; url - php=hhvm
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
 ireplay.fr/text/..iReplay/1.0 (url) - php=hhvm
simplepie
metamagazine
 metamagazine.comtext/..WordPress/3.9.2; url -
tt-rss
scarecrow
 www.scarecrow.com/application/jsonScarecrow/1.0 (url; mail address ) - php=hhvm
 www.scarecrow.com/application/jsonScarecrow/1.0 (url; mail address ) - php=zend
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - php=hhvm
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - php=hhvm
testomato
 www.testomato.com/testomatobottext/..Testomatobot/1.0 (Linux x86_64; url) minicrawler/3.0~alpha en-US,en,cs;q=0.01 php=hhvm
 www.testomato.com/testomatobottext/..Testomatobot/1.0 (Linux x86_64; url) minicrawler/3.0~beta en-US,en,cs;q=0.01 php=hhvm
 www.testomato.com/testomatobottext/..Testomatobot/1.0 (Linux x86_64 url) minicrawler/3.0~beta - php=hhvm
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - php=hhvm
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - php=hhvm
cdlib
 webarchives.cdlib.org/p/webmasterstext/..Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 url) - php=hhvm
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - php=hhvm
 www.thinglink.com/help/ThinglinkImageBottext/..Elmer, the Thinglink ImageBot (url) - php=hhvm
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - php=zend
 www.thinglink.com/help/ThinglinkImageBotimage/..Elmer, the Thinglink ImageBot (url) - -
loopj
 loopj.com/android-async-httpapplication/jsonandroid-async-http/1.4.4 (url) - php=hhvm
inagist
 inagist.comtext/..InAGist URL Resolver (url) en-us,en;q=0.5 php=hhvm
grammarly
 www.grammarly.comapplication/jsonGrammarly/1.0 (url) - php=hhvm
similartech
 www.similartech.com/smtbottext/..Mozilla/5.0 (compatible; SMTBot/1.0; url) - php=hhvm
psu
 citeseerx.ist.psu.edu/image/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-2014-11-14T15:29:34Z url) - -
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - php=hhvm
sciencecard
 dryad.sciencecard.orgapplication/jsonArticle-Level Metrics 3.6.3 - url - php=hhvm
crowsnest
 www.crowsnest.tv/text/..Crowsnest/0.5 (url) - php=hhvm
egerin
 www.egerin.comtext/..crawler (url); Kurdish search engine ku php=hhvm
bestonlinesongs
 www.bestonlinesongs.com/text/..MyBot/1.0 (url) - php=hhvm
 www.bestonlinesongs.com/text/..MyBot/1.0 (url) - php=zend
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja php=hhvm
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja php=hhvm;https=1
site-shot
 www.site-shot.com/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Chromium/31.0.1650.63 Site-Shot/2.1 (url) Safari/534.34 en-US,* -
 www.site-shot.com/text/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Chromium/31.0.1650.63 Site-Shot/2.1 (url) Safari/534.34 en-US,* php=hhvm
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address ) - php=hhvm
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - php=hhvm
knowly
 www.knowly.net/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - php=hhvm
 www.knowly.net/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - php=zend
diffbot
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 php=hhvm
perfectmarket
 www.perfectmarket.comtext/..Mozilla/5.0 (compatible; PerfectMarketKWTbot/1.0 url) - php=hhvm
 www.perfectmarket.comtext/..Mozilla/5.0 (compatible; PerfectMarketKWTbot/1.0 url) - php=zend
mynameisgladiator
 www.mynameisgladiator.comtext/..WordPress/3.3.2; url - php=hhvm
image2play
 image2play.com/image/..Mozilla/5.0 (compatible; Image2play/0.1; url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 php=hhvm
thecardb
 thecardb.comtext/..theCarDB/1.1 (url; mail address ) - php=hhvm
 thecardb.comtext/..theCarDB/1.1 (url; mail address ) - php=zend
ac
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - php=hhvm
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) - php=hhvm
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) - php=zend
creativepulses
 creativepulses.nlapplication/jsonCreativePulses Crawler (url) - php=hhvm
 creativepulses.nlapplication/jsonCreativePulses Crawler (url) - php=zend
popularmusicvid
 www.popularmusicvid.com/text/..MyBot/1.0 (url) - php=hhvm
havos
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - php=hhvm
 www.picsearch.com/bot.htmltext/..psbot-page (url) - php=hhvm
 www.picsearch.com/bot.htmltext/..psbot-image (url) - php=hhvm
utu
 bionlp.utu.fitext/..Python-urllib/2.7 (url) - php=hhvm
zmescience
 www.zmescience.comtext/..WordPress/4.0.1; url - php=hhvm
wikidata
 www.wikidata.orgapplication/jsonurl - php=hhvm
livelap
 site.livelap.com/crawlertext/..LivelapBot/0.2 (url) en-US,en php=hhvm
 site.livelap.com/crawlerimage/..LivelapBot/0.2 (url) en-US,en -
QtWeb
joka02
 joka02.infoimage/..WordPress/4.1; url - -
17daydiets
 www.17daydiets.comtext/..WordPress/4.0.1; url - php=hhvm
fredtrotter
 www.fredtrotter.com/application/jsonClincalSpade/1.0 (url; mail address ) - php=hhvm
blogspot
 alberapps.blogspot.comtext/..TiempoBus/3.1.4 (url; mail address ) - php=hhvm
 alberapps.blogspot.comtext/..TiempoBus/3.1.4 (url; mail address ) - php=hhvm;https=1
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - php=hhvm
comingupapp
 comingupapp.com/text/..ComingUp/3.0 (url; mail address ) - php=hhvm
188034.399999978total

Page requests for probable crawlers, recognized by keyword
Count
x 1000
Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - php=hhvm
 application/vnd.php.serialized
 text/..
 application/json
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/pdf
wikiwix-bot-3.0 - php=hhvm
 text/..
php wikibot classes - php=hhvm
 application/vnd.php.serialized
 application/json
 text/..
MediaWikiCrawler-Google - php=hhvm
 text/..
 application/json
LinkParser/2.0 - php=hhvm
 text/..
ClueBot/2.0 - php=hhvm
 application/vnd.php.serialized
 text/..
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - php=zend
 application/vnd.php.serialized
 text/..
 application/json
GoogleBot-Image/1.0 - php=hhvm
 text/..
 image/..
py-radehh.py/rg11106 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
RBot/0.3 ( mail address ) -
 application/json
 image/..
 text/..
py-radehh.py/rg11106 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN php=hhvm
 text/..
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - php=hhvm
 application/json
 text/..
strucr.com crawler 3.25.410 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=hhvm
 text/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
welcome (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
pwb/rg3113 Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
welcome (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
Mozilla/5.0 (compatible; OrangeBot/2.0; mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 php=hhvm
 text/..
pwb/rg3113 Pywikipediabot/2.0 - php=hhvm
 application/json
LinkParser/2.0 - php=zend
 text/..
php wikibot classes - php=zend
 application/vnd.php.serialized
 text/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
AniBot/0.9 php/curl - php=hhvm
 application/vnd.php.serialized
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
WikiGameBot/0.2 ( mail address ) - php=hhvm
 application/json
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
 text/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - php=hhvm
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm
 text/..
 application/xml
py-radehh.py/rg11106 Pywikipediabot/1.0 - php=zend
 application/json
maj_articles_recents (wikipedia:fr; User:ZéroBot) Pywikibot/2.0b3 (g4795) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
Twitterbot/1.0 - php=hhvm
 text/..
 application/json
spider - php=hhvm
 text/..
 application/json
 application/vnd.php.serialized
pywikibot-1newar25.py/rg11106 Pywikipediabot/1.0 - php=zend
 application/json
maj_articles_recents (wikipedia:fr; User:ZéroBot) Pywikibot/2.0b2 (g4249) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; php=hhvm
 text/..
ClueBot/2.0 - php=zend
 application/vnd.php.serialized
MediaWiki::Bot/5.005006 - php=hhvm
 application/json
MediaWiki::Bot/3.2.6 - php=hhvm
 application/json
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-zzarticle_importance_talk.py/rg11084 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
wymowa (commons:commons; User:Alkamid) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
tigerbot - php=hhvm;https=1
 application/json
 text/..
wiktionaryInterwiki (wiktionary:mg; User:Bot-Jagwar) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
MediaWikiCrawler-Google/2.0 ( mail address ) - php=zend
 text/..
DotNetWikiBot/3.11 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm
 text/..
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - php=hhvm;https=1
 application/json
 text/..
welcome (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
 text/..
tigerbot - php=hhvm
 text/..
 application/json
DigitalsmithsBot - php=hhvm
 text/..
pywikipedia-git-ul_wd.py/rg90 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
wymowa (commons:commons; User:Alkamid) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
strucr.com crawler 3.25.403 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=hhvm
 text/..
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - php=hhvm
 application/json
 text/..
pwb/rg3113 Pywikipediabot/2.0 - php=zend
 application/json
 text/..
strucr.com crawler 3.25.406 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=hhvm
 text/..
MediaWikiCrawler-Google - php=zend
 text/..
MediaWikiCrawler-Google/2.0 ( mail address ) - php=hhvm
 text/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-radeh.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
wiktionaryInterwiki (wiktionary:mg; User:Bot-Jagwar) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
pywikipedia-radeh.py/rg11084 Pywikipediabot/1.0 - php=hhvm
 application/json
DotNetWikiBot/3.11 (Microsoft Windows NT 6.2.9200.0; ) - php=hhvm
 text/..
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - php=hhvm
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm;https=1
 text/..
 application/xml
addLabel (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
Owlin Bot v. 3.0 - php=hhvm
 text/..
 -
 application/xml
pywikipedia-radeh.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
WorldOfMusicBetaBot/1.0 - php=hhvm
 text/..
PythonWikipediaBot/1.0 - php=hhvm
 application/json
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - php=hhvm
 application/json
wikiwix-bot-3.0 - php=zend
 text/..
maj_articles_recents (wikipedia:fr; User:ZéroBot) Pywikibot/2.0b2 (g4249) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11106 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
 application/pdf
addbot (commons:commons; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
python-wikitools/1.2 (User:DickensBot) -
 application/json
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - php=zend
 application/json
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
AniBot/0.9 php/curl - php=zend
 application/vnd.php.serialized
MediaWiki::Bot/3.1.6 (User:SporkBot) - php=hhvm
 application/json
inspect/r2160 Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
LTBot en-us,en;q=0.5
 text/..
 application/xml
py-radehh.py/rg10975 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
py-navbox.py/rg11106 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
pywikibot-1newar25.py/rg11108 Pywikipediabot/1.0 - php=hhvm
 application/json
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - php=zend
 application/json
 text/..
MediaWiki::Bot/3.005002 - php=hhvm
 application/json
pywikibot-1newar25.py/rg11111 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
UAA-UAA.py/r10417 Pywikipediabot/1.0 - php=hhvm
 application/json
UAA-UAA.py/r10417 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
ContextAd Bot 1.0 - php=hhvm
 text/..
 application/json
MediaWiki::Bot/3.2.6 - php=zend
 application/json
 text/..
Screaming Frog SEO Spider/2,55 - php=hhvm
 text/..
iw2data/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
strucr.com crawler 3.25.410 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=hhvm;https=1
 text/..
addbot (commons:commons; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
pywikibot-1newar25.py/rg11115 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
pywikipedia-zz_make_cat.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
HTMLParser/2.0 - php=hhvm
 text/..
iw2data/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
pywikipedia-newinterwiki-fa.py/rg11077 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
MediaWiki::Bot/5.005006 - php=zend
 application/json
addbot (wikidata:wikidata; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
graydonCrawler/Nutch-1.9 (Graydon crawler, for testing purposes only; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - php=zend
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* php=hhvm
 text/..
 image/..
 application/json
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
pywikipedia-newinterwiki-fa.py/rg11077 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
GoogleBot-Image/1.0 - php=hhvm
 text/..
 application/json
 image/..
pywikipedia-zzfatranslator.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
Twitterbot/1.0 - -
 image/..
 text/..
 application/pdf
 application/ogg
renametemp (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
tigerbot - php=zend
 application/json
 text/..
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - php=hhvm
 text/..
mrmetadata (wikipedia:en) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - php=hhvm
 application/json
node-crawler/0.2.5 - php=hhvm
 text/..
mussol-mussol.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
Webbot - php=hhvm
 text/..
GermCrawler - php=hhvm
 application/json
 text/..
 application/xml
charge2 (wikidata:wikidata; User:Thieolbot) Pywikibot/2.0b3 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm;https=1
 application/json
charge3 (wikidata:wikidata; User:Thieolbot) Pywikibot/2.0b3 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm;https=1
 application/json
addbot (wikidata:wikidata; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b3 (3) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
pywikipedia-zzarticle_importance_talk.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - php=hhvm;https=1
 text/..
 application/xml
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - php=hhvm
 text/..
 application/xml
GoogleBot (gocrawl v0.4) - php=hhvm
 text/..
AnomieBOT/1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - php=hhvm
 application/json
pywikibot-compat-commonscat.py/rg11115 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
robospider/1.0 - php=hhvm
 text/..
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
es-pp (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm;https=1
 application/json
wymowa (commons:commons; User:Alkamid) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
DigitalsmithsBot - php=zend
 text/..
Nutch Spider/Nutch-2.2.1 - php=hhvm
 text/..
UAA-UAA.py/r10417 Pywikipediabot/1.0 - php=zend
 application/json
SineBot/1.5.19(User:SineBot) - php=hhvm
 application/vnd.php.serialized
 text/..
Screaming Frog SEO Spider/2.55 - php=hhvm
 text/..
tdrx-location-image-crawler: one-time crawl, about 100,000 articles; please contact mail address and temporarily block if this causes any inconvenience to the site. I am using a single thread. - php=hhvm
 text/..
compat-label_bot.py/rg11051 Pywikipediabot/1.0 - php=hhvm
 application/json
strucr.com crawler 3.25.402 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=hhvm
 text/..
WikiGameBot/0.1 ( mail address ) - php=hhvm
 application/json
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
interwiki (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
dikantenyvaovao (wiktionary:en) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
compat-checkimages.py/rs11085 Pywikipediabot/1.0 - php=hhvm
 application/json
WikidataBot framework - php=hhvm
 application/json
Spiderbot/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
mrmetadata (wikipedia:en) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
MyCuteBot/0.1 - php=hhvm
 text/..
 application/json
compat-checkimages.py/rs11090 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
ko_missing (wikipedia:en; User:Chobot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
cat_finder-core (wikipedia:fa; User:Rezabot) Pywikibot/2.0b2 (g4582) httplib2/0.8 Python/2.7.3.final.0 - php=hhvm
 application/json
GoogleBot-Image/1.0 - https=1;php=hhvm
 text/..
cat_finder-core (wikipedia:fa; User:Rezabot) Pywikibot/2.0b2 (g4582) httplib2/0.8 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
iw2data/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN php=hhvm;https=1
 text/..
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - php=hhvm
 application/json
 text/..
pywikipedia-newinterwiki-fa.py/rg11077 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-zzfatranslator.py/rg11084 Pywikipediabot/1.0 - php=hhvm
 application/json
interwiki (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
BoodiBot/0.0.0; https://boodigo.com/) - php=hhvm
 text/..
addLabelFromOtherLanguage/r2916 Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN php=zend
 text/..
 application/xml
compat-redirect.py/rs11086 Pywikipediabot/1.0 - php=hhvm
 application/json
 text/..
maj_articles_recents (wikipedia:fr; User:ZéroBot) Pywikibot/2.0b2 (g4249) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
spider - php=zend
 text/..
 application/json
AnomieBOT/1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - php=hhvm
 application/json
DotNetWikiBot/3.11 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
GoogleBot-Image/1.0 - php=zend
 text/..
py-radehh.py/rg10975 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
compat-redirect.py/rs11090 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 text/..
mrmetadata (commons:commons; User:Seven-League_Bot) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b2 (3) httplib2/0.9 Python/2.7.8.final.0 - php=hhvm
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - php=hhvm;https=1
 application/json
TicketsBot/0.1 - php=hhvm
 text/..
WorldOfMusicBetaBot/1.0 - php=zend
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - php=hhvm
 application/json
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) - php=hhvm
 text/..
ifd-empty (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - php=hhvm
 text/..
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - php=zend
 application/json
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - php=hhvm
 application/json
strucr.com crawler 3.25.405 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=hhvm
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - php=hhvm
 application/json
pywikibot-1newar25.py/rg11114 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - php=zend
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.60; ) - php=hhvm
 text/..
 application/xml
strucr.com crawler 3.25.404 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=hhvm
 text/..
inflectionof (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
welcome (wikipedia:hr; User:SpeedyBot) Pywikibot/2.0b3 (g4670) httplib2/0.9 Python/2.6.6.final.0 - php=hhvm
 application/json
Omnilibbot/1.0 en, fr,ro,* php=hhvm
 text/..
welcome (wikipedia:ru; User:Dima_st_bk_bot) Pywikibot/2.0b3 (g3) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
pywikipedia-git-desc_bot5.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.3; zh-CN; SCH-N719 Build/JSS15J) AppleWebKit/533.1 KHTML Version/4.0 YisouSpider/1.0.0 Mobile Safari/533.1 zh-cn,en-us,zh-tw,en-gb,en; php=hhvm
 text/..
cat_finder-core (wikipedia:fa; User:Rezabot) Pywikibot/2.0b2 (g4582) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
viaf_import (wikidata:wikidata; User:BotMultichill) Pywikibot/2.0b3 (g4818) httplib2/0.8 Python/2.7.2.final.0 - php=hhvm;https=1
 application/json
AnomieBOT/1.0 (TemplateSubster; see [[User:AnomieBOT]]) - php=hhvm
 application/json
YisouSpider - php=hhvm
 text/..
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; php=hhvm;https=1
 text/..
Bot - php=hhvm
 text/..
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - php=hhvm
 application/json
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - php=hhvm
 application/vnd.php.serialized
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
SentiBot www.sentibot.eu (compatible with GoogleBot) - php=hhvm
 text/..
renametemp (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11117 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
py-navbox.py/rg11106 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
DotNetWikiBot/3.11 (Microsoft Windows NT 6.2.9200.0; ) - php=hhvm;https=1
 text/..
mrmetadata (commons:commons; User:Seven-League_Bot) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
SPIupdater-globalfunc.py/r10417 Pywikipediabot/1.0 - php=hhvm
 application/json
compat-label_bot.py/rg11051 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - php=zend
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - php=hhvm;https=1
 text/..
 application/xml
MediaWiki::Bot/3.005002 - php=zend
 application/json
pywiki-newar25.py/rg11115 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
milog_bot/1.0 ( mail address ) - php=hhvm
 text/..
pywikibot-compat-commonscat.py/rg11110 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
GoogleBot-Video/1.0 - php=hhvm
 text/..
cat_finder-core (wikidata:wikidata; User:Rezabot) Pywikibot/2.0b2 (g4582) httplib2/0.8 Python/2.7.3.final.0 - php=hhvm
 application/json
User-Agent Baiduspider - php=hhvm
 text/..
 image/..
dikantenyvaovao (wiktionary:en) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
run_replace.7r.02/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
inflectionof (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
SearchBot/Nutch-1.9 en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
 application/xml
pywikibot-1newar25.py/rg11120 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
pywikipedia-zzfatranslator.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
cat_finder-core (wikidata:wikidata; User:Rezabot) Pywikibot/2.0b2 (g4582) httplib2/0.8 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
WikiGameBot/1.1 ( mail address ) - php=hhvm;https=1
 application/json
gnd (wikipedia:de) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - php=hhvm
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
pywikipedia-git-com10.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
cat_finder-core (wikidata:wikidata; User:Rezabot) Pywikibot/2.0b2 (g4582) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - php=hhvm
 text/..
gnd (wikipedia:de) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
charge4 (wikidata:wikidata; User:Thieolbot) Pywikibot/2.0b3 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm;https=1
 application/json
YandexExtdataBot/1.0 ( mail address ) - php=hhvm
 application/json
minicrawler/3.0~alpha - php=hhvm
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 php=hhvm
 text/..
refreshfeeds (wikinews:ru; User:AKBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
pywikipedia-git-sectionBoten.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
Mozilla/5.0 (Linux; Android/4.3; zh-CN; SCH-N719 Build/JSS15J) AppleWebKit/533.1 KHTML Version/4.0 YisouSpider/1.0.0 Mobile Safari/533.1 - php=hhvm
 text/..
My Nutch Spider/Nutch-2.2.1 - php=hhvm
 text/..
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - php=hhvm;https=1
 text/..
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm
 text/..
 application/xml
dikantenyvaovao (wiktionary:mg; User:Bot-Jagwar) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
addbot (wikipedia:en; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
Mozilla/5.0 (Linux; Android/4.3; ru-ru; Spider Build/JLS36C) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 ru-RU, en-US -
 image/..
GoogleBot/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
WikidataBot framework - php=zend
 application/json
pywikibot-1newar25.py/rg11107 Pywikipediabot/1.0 - php=zend
 application/json
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - php=zend
 application/json
PythonWikipediaBot/1.0 - php=zend
 application/json
 text/..
dikantenyvaovao (wiktionary:mg; User:Bot-Jagwar) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - php=hhvm
 application/json
Mozilla/5.0 (Apibot Browser) - -
 image/..
 application/ogg
strucr.com crawler 3.25.402 (refer to in robots.txt as strucr, see https://strucr.com/bot) - php=zend
 text/..
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - php=zend
 application/json
wymowa (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
pywikipedia-git-P132_fixer.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
SPIupdater-globalfunc.py/r10417 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
Goalkeeperbot(User:Beetstra)/1.0 - php=hhvm
 text/..
XLinkBot/1.00 - php=hhvm
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - php=zend
 application/json
 text/..
pywikipedia-git-desc_bot5.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
Bot at mysite.com - php=hhvm
 text/..
interwiki (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
clean_sandbox (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
compat-interwiki.py/rg11111 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us
 image/..
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
PHPCrawl - php=hhvm
 text/..
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - php=hhvm;https=1
 text/..
addbot (wikipedia:en; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
SPIupdater-globalfunc.py/r10417 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
sport (wikipedia:fr; User:NaggoBot) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
bot-07-fix-language-mappings (wikidata:wikidata; User:AkkakkBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm;https=1
 text/..
 application/xml
German Wikipedia Broken Weblinks Bot; contact: mail address - php=hhvm
 text/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
 application/xml
pywikibot-compat-aaa_categorieAutori.py/rg10952 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
DotNetWikiBot/3.11 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
Twitterbot/1.0 - php=hhvm;https=1
 text/..
redirect (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
welcome-c (wikipedia:fr; User:Loveless) Pywikibot/2.0b2 (g4566) httplib2/0.9 Python/2.7.6.final.0 - php=hhvm;https=1
 application/json
minicrawler/3.0~beta - php=hhvm
 text/..
addDescCat/r2916 Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
php wikibot classes - php=hhvm;https=1
 application/vnd.php.serialized
Twitterbot/1.0 - php=zend
 text/..
run_replace.7r.02/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
adddescralbum (wikidata:wikidata; User:RobotMichiel1972) Pywikibot/2.0b3 (s5895) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - php=hhvm
 text/..
 application/xml
pwb/rg3009 Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.60; ) - php=hhvm;https=1
 text/..
 application/xml
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - php=zend
 application/json
redirect (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
wymowa (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
GoogleBot-Video/1.0 - -
 application/ogg
 text/..
 video/webm
 audio/x-flac
 audio/webm
 audio/midi
run_replace.7r.03/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
pwb/rg3009 Pywikipediabot/2.0 - php=hhvm
 application/json
py-interwiki.py/rg11047 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
gnd2/r2916 Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
pywikibot-compat-commonscat.py/rg11113 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
CorenSearchBot/1.7 en libwww-perl/6.05 - php=hhvm
 text/..
CorenSearchBot/1.7 en libwww-perl/6.05 - php=hhvm;https=1
 text/..
run_replace.7r.03/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm
 application/json
welcome-c (wikipedia:fr; User:Loveless) Pywikibot/2.0b2 (g4566) httplib2/0.9 Python/2.7.6.final.0 - php=hhvm
 application/json
AnomieBOT/1.0 (BAGBot; see [[User:AnomieBOT]]) - php=hhvm
 application/json
 text/..
6680957 (wikipedia:en; User:RussBot) Pywikibot/2.0b3 (g2) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
clean_sandbox (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
pywikipedia-zzaddnavebox.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - php=hhvm
 application/json
 application/xml
pywikibot-compat-commonscat.py/rg11111 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - php=zend
 application/json
gnd2 (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.5.final.0 - php=hhvm;https=1
 application/json
Mozilla/5.0 (compatible; OrangeBot/2.0; mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 php=zend
 text/..
python-wikitools/1.2 (User:Irclogbot) - php=hhvm;https=1
 application/json
com.apple.Safari.WebFeedParser/600.1.4 CFNetwork/711.1.16 Darwin/14.0.0 en-us php=hhvm
 application/xml
 text/..
bot-07-fix-language-mappings (wikidata:wikidata; User:AkkakkBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
ifd-empty (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
GermCrawler - php=zend
 application/json
 application/xml
 text/..
compat-basictemplatedisamblist.py/rg11090 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
MediaWiki::Bot/3.1.6 (User:SporkBot) - php=zend
 application/json
pywikipedia-git-com9.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
BeneBot*/1.0 WikibasePhpLib/0.1 - php=zend
 application/json
inspectMissingTerms/rg3599 Pywikibot/2.0 - php=hhvm
 application/json
GoogleBot-Image/1.0 - php=hhvm;https=1
 text/..
dikantenyvaovao (wiktionary:ru) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
compat-checkimages.py/rs11096 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
MyCuteBot/0.1 - php=zend
 text/..
 application/json
ifd-hoobot (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
compat-checkimages.py/rs11085 Pywikipediabot/1.0 - php=zend
 application/json
gnd (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Peachy MediaWiki Bot API Version 1.0 - php=hhvm
 application/vnd.php.serialized
wikiparser/1 CFNetwork/673.4 Darwin/13.4.0 (x86_64) (MacPro5,1) en-us
 image/..
 text/..
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
 text/..
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" - -
 image/..
 text/..
compat-basicarticledisamblist.py/rg11090 Pywikipediabot/1.0 - php=zend
 application/json
Webwiki Search Engine Bot - www.webwiki.de - php=hhvm
 text/..
sport (wikipedia:fr; User:NaggoBot) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - php=hhvm
 application/json
P107ruwiki/r2916 Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
gnd2/r2916 Pywikipediabot/2.0 - php=hhvm
 application/json
NutchSpider/Nutch-1.10-SNAPSHOT (Nutch crawler used for research purpose at Curtin university.) en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
adddescr (wikidata:wikidata; User:RobotMichiel1972) Pywikibot/2.0b3 (s5895) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
MediaWiki::Bot/5.005006 (User:lustiger_seth) - php=hhvm;https=1
 application/json
pywikipedia-zzpurging.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
CorenSearchBot/1.7 en libwww-perl/6.05 - php=zend
 text/..
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - php=hhvm
 text/..
FP (commons:commons; User:SteinsplitterBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
replace (wikipedia:ca; User:KRLS_Bot) Pywikibot/2.0b2 (g4609) httplib2/0.9 Python/2.7.6.final.0 - php=hhvm
 application/json
LTBot en-us,en;q=0.5 php=hhvm;https=1
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; fr-fr; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 fr-FR, en-US -
 image/..
refreshfeeds (wikinews:ru; User:AKBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
FP (commons:commons; User:SteinsplitterBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.3.final.0 - php=hhvm
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
LRI.fr bot mail address - php=hhvm
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
BotReversor-BotReversor.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
gnd (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
SineBot/1.5.19(User:SineBot) - php=zend
 application/vnd.php.serialized
 text/..
milog_bot/1.0 ( mail address ) - php=zend
 text/..
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - php=hhvm
 text/..
BOT/0.1 (BOT for JCE) - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
AnomieBOT/1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - php=zend
 application/json
compat-label_bot.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
pywiki-lit.py/rg11115 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
a (wikipedia:mr; User:निनावी) Pywikibot/2.0b2 httplib2/0.9 Python/3.4.2.final.0 - php=hhvm
 application/json
pywikipedia-zzarticle_importance.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
ifd-hoobot (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
mrmetadata (wikipedia:de) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
mrmetadata (wikipedia:de) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
dikantenyvaovao (wiktionary:en) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
cd-delinker.py/r11775 Pywikipediabot/1.0 - php=hhvm
 application/json
insertDistriWD (wikidata:wikidata; User:Symac_bot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
UKBot [[:no:Bruker:UKBot]] - MwClient/0.7.2dev (https://github.com/mwclient/mwclient) - php=hhvm
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
COIBot/1.00 - php=hhvm
 text/..
COIBot/2.0 - php=hhvm
 text/..
redirect (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b2 (3) httplib2/0.9 Python/2.7.8.final.0 - php=zend
 application/json
6277263 (wikipedia:en; User:RussBot) Pywikibot/2.0b3 (g3) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-zzarticle_importance_talk.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
GoogleBot-Image/1.0 -
 text/..
 image/..
dikantenyvaovao (wiktionary:en) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-zzpurging.py/rg11084 Pywikipediabot/1.0 - php=hhvm
 application/json
PerlWikiBot/5.006002 (https://ca.wikipedia.org/wiki/Usuari:Langtoolbot; User:Jaumeortola) - php=hhvm;https=1
 application/json
refreshfeeds (wikinews:ru; User:AKBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-newar.py/rg11106 Pywikipediabot/1.0 - php=zend
 application/json
JackBot-fr.wikt.RemiseEnForme.py/r96 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
addCoordFromTemplate/r2916 Pywikipediabot/2.0 - php=hhvm
 application/json
node.js - nodemw - mail address - Maintainer: Rainer Rillke - mail address - [[:commons:User:Rillke]] - php=hhvm
 application/json
MoodleBot/1.0 - php=hhvm
 application/xml
 text/..
 application/vnd.php.serialized
addbot (wikipedia:de; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
addbot (commons:commons; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywiki-cfd.py/rg11032 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-zzaddnavebox.py/rg11084 Pywikipediabot/1.0 - php=hhvm
 application/json
pywikibot-1newar25.py/rg11107 Pywikipediabot/1.0 - php=hhvm
 application/json
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - php=hhvm
 application/xml
 text/..
py-navbox.py/rg11106 Pywikipediabot/1.0 - php=zend
 application/json
SurakWare MediaWiki Bot/1.0 - php=hhvm;https=1
 text/..
pywikibot-compat-commonscat.py/rg11120 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm
 application/json
shwikiCoor (wikipedia:sh) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - php=zend
 application/json
AnomieBOT/1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - php=zend
 application/json
adddescralbum (wikipedia:en) Pywikibot/2.0b3 (s5895) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
compat-redirect.py/rg10957 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
WikiBot/0.1 - php=hhvm
 text/..
clean_sandbox (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - php=hhvm
 application/json
geonames (wikipedia:sh) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
indexer2 (wikipedia:en; User:Legobot) Pywikibot/2.0b3 (g4842) httplib2/0.9 Python/3.4.0.final.0 - php=hhvm;https=1
 application/json
pywikibot-compat-dobrodoslice.py/rg11089 Pywikipediabot/1.0 - php=hhvm
 application/json
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - php=zend
 application/json
 text/..
welcome-c (wikipedia:fr; User:Loveless) Pywikibot/2.0b2 (g4566) httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
bot-07-fix-language-mappings (wikidata:wikidata; User:AkkakkBot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
dikantenyvaovao (wiktionary:ru) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
pywikibot-compat-commonscat.py/rg11117 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
pywikibot-compat-commonscat.py/rg11119 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
cities (wikipedia:en) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm;https=1
 application/json
DotNetWikiBot/2.92 (Microsoft Windows NT 6.1.7600.0; ) - php=hhvm
 text/..
 application/xml
WPendata-importcategoryDEFAULT.py/r11052 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
redirect (wikipedia:en; User:Xqbot) Pywikibot/2.0b3 (s5756) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
mrmetadata (wikipedia:it) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - php=hhvm;https=1
 text/..
 application/xml
MoodleBot/1.0 -
 application/xml
 text/..
 application/vnd.php.serialized
 image/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm
 text/..
DotNetWikiBot/2.95 (Microsoft Windows NT 6.2.9200.0; ) - php=hhvm
 text/..
addbot (wikidata:wikidata; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - php=hhvm
 text/..
 application/xml
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
mrmetadata (wikipedia:ru) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
compat-redirect.py/rs11096 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 text/..
SurakWare MediaWiki Bot/1.0 - php=hhvm
 text/..
 application/xml
DotNetWikiBot/2.96 (Microsoft Windows NT 6.2.9200.0; ) - php=hhvm
 text/..
Owlin Bot v. 3.0 - php=zend
 text/..
aranhabot - -
 image/..
delete_template (wikipedia:ru; User:WinterheartBot) Pywikibot/2.0b3 (g4666) httplib2/0.9 Python/3.3.5.final.0 - php=hhvm;https=1
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - php=zend
 text/..
JackBot-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
DotNetWikiBot/3.11 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm;https=1
 text/..
addClaimFromTemplate/r2916 Pywikipediabot/2.0 - php=zend
 application/json
pwb/rg3009 Pywikipediabot/2.0 - php=zend
 application/json
Wikibot/2.1.1 CFNetwork/711.1.16 Darwin/14.0.0 en-us -
 image/..
 text/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - php=hhvm
 text/..
 application/xml
 application/json
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
Projet Sport-Projetsportliste2.py/r11052 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
CaBot - php=hhvm
 text/..
welcome (wikipedia:lt; User:Homobot) Pywikibot/2.0b2 (3) httplib2/0.9 Python/2.7.8.final.0 - php=hhvm;https=1
 application/json
welcome (wikipedia:hr; User:SpeedyBot) Pywikibot/2.0b3 (g4670) httplib2/0.9 Python/2.6.6.final.0 - php=hhvm;https=1
 application/json
MediaWiki::Bot/5.005006 (User:lustiger_seth) - php=hhvm
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; php=zend
 text/..
BotReversor-BotReversor.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm
 application/json
sport (wikipedia:fr; User:NaggoBot) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
AnomieBOT/1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - php=hhvm
 application/json
mrmetadata (wikipedia:ru) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - php=zend
 application/vnd.php.serialized
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
pywikipedia-commonscat.py/rg11117 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
pywiki-cfd.py/rg11032 Pywikipediabot/1.0 - php=hhvm
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
python (wikipedia:cs; User:Frettiebot) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
pywikipedia-zziwneededpage.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
pywikibot-compat-dobrodoslice.py/rg11089 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
FP (commons:commons; User:SteinsplitterBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-zzgallery.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
t9/rg2816 Pywikipediabot/2.0 - php=hhvm;https=1
 application/json
addbot (wikipedia:de; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
pywikipedia-zzpurging.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.3; Spider Build/JLS36C) AppleWebKit/537.36 KHTML Chrome/39.0.2171.93 Mobile Safari/537.36 ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4 -
 image/..
mrmetadata (wikipedia:it) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
PythonWikipediaBot/1.0 -
 application/json
 text/..
AnomieBOT/1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - php=hhvm
 application/json
mrmetadata (wikipedia:en) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
redirect/-1 (unknown) Pywikibot/2.0 (User:Rahl43bot) - php=hhvm
 application/json
My Nutch Spider/Nutch-1.9 en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
Mozilla/5.0 (compatible; SputnikImageBot/2.2) - -
 image/..
 text/..
AnomieBOT/1.0 (TemplateSubster; see [[User:AnomieBOT]]) - php=zend
 application/json
compat-redirect.py/rs11076 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - php=zend
 text/..
 application/xml
compat-pwb.py/rg11092 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
 application/xml
 text/..
Yeobot/1.1 Contact: mail address ) - php=hhvm
 text/..
py-interwiki.py/rg11047 Pywikipediabot/1.0 - php=hhvm
 application/json
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - php=zend
 application/json
CaBot Script (running on nightshade.toolserver.org) - php=hhvm
 application/vnd.php.serialized
ContextAd Bot 1.0 - php=zend
 text/..
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - php=zend
 application/json
compat-checkimages.py/rs11084 Pywikipediabot/1.0 - php=zend
 application/json
vm-auto-erl-neu (wikipedia:de; User:Xqbot) Pywikibot/2.0b3 (s5704) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
pywikibot-1newar25.py/rg11110 Pywikipediabot/1.0 - php=hhvm
 application/json
pywikipedia-newar.py/rg11108 Pywikipediabot/1.0 - php=hhvm
 application/json
YOUBot (Youniversal Media Bot) - php=hhvm
 text/..
replace (wikipedia:ca; User:KRLS_Bot) Pywikibot/2.0b2 (g4609) httplib2/0.9 Python/2.7.6.final.0 - php=hhvm;https=1
 application/json
compat-pwb.py/rg11092 Pywikipediabot/1.0 - php=hhvm
 application/json
 text/..
 application/xml
pywikipedia-interwiki.py/rg11 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
ConBot - php=hhvm
 text/..
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-wikifyer.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
geonames (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
TicketsBot/0.1 - php=zend
 text/..
RSS Bot/1.9 (Mac OS X Version 10.10.1 (Build 14B25)) en;q=1 php=hhvm
 application/xml
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlBot) - php=hhvm
 text/..
German Wikipedia Broken Weblinks Bot; contact: mail address - php=zend
 text/..
OpenSearchServer_Bot - php=hhvm
 text/..
pywikipedia-zziwneededpage.py/rg11084 Pywikipediabot/1.0 - php=hhvm
 application/json
pywikipedia-zzfawikipatroler.py/rg11084 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
mrmetadata (wikipedia:id) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
pywikibot-compat-redirect.py/rg10993 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
Mozilla/5.0 (compatible; MojeekBot/0.6; https://www.mojeek.com/bot.html) - php=hhvm
 text/..
dikantenyvaovao (wiktionary:mg; User:Bot-Jagwar) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=hhvm
 application/json
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-newar.py/rg11106 Pywikipediabot/1.0 - php=hhvm
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - php=zend
 text/..
Peachy MediaWiki Bot API Version 1.0 - php=zend
 application/vnd.php.serialized
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
BOT/0.1 (BOT for JCE) - php=hhvm
 text/..
pywikibot-compat-dobrodoslice.py/rg11089 Pywikipediabot/1.0 - php=zend
 application/json
compat-basicarticledisamblist.py/rg11090 Pywikipediabot/1.0 - php=hhvm
 application/json
pywikipedia-lit.py/rg11106 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-git-cat_rd_wd.py/rg90 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
mrmetadata (wikipedia:he) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
compat-checkimages.py/rs11085 Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
SurakWare MediaWiki Bot/1.0 - php=zend
 text/..
RokaSpider/Nutch-1.9 (' mail address ') de-de,de,en-us,en-gb,en;q=0.7,*;q=0.3 php=hhvm
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
ifd-empty (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
PythonWikipediaBot/1.0 - php=hhvm;https=1
 text/..
 application/json
compat-redirect.py/rs11085 Pywikipediabot/1.0 - php=zend
 application/json
mrmetadata (wikipedia:fr) Pywikibot/2.0b3 httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - php=hhvm
 text/..
adddescrauteur (wikidata:wikidata; User:RobotMichiel1972) Pywikibot/2.0b3 (s5895) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm;https=1
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - php=zend
 text/..
pywikipedia-git-featured_new.py/rg90 Pywikipediabot/1.0 - php=hhvm
 application/json
 text/..
ifd-hoobot (wikidata:wikidata; User:PLbot) Pywikibot/2.0b3 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
addbot (wikipedia:en; User:YiFeiBot) Pywikibot/2.0b2 (g4635) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikilab-llista_mils.py/r-1 (unknown) Pywikipediabot/1.0 - php=hhvm;https=1
 application/json
cdh-delinker.py/r11775 Pywikipediabot/1.0 - php=hhvm
 application/json
GoogleBot-Image/1.0 - php=zend
 text/..
 application/json
34435.97total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Thu, Jan 8, 2015 20:51
Author:Erik Zachte (
Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers