Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Sep 2014 - 30 Sep 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 159,822,530 page requests (mime type text/html only!) per day are considered crawler requests, out of 673,008,800 external requests, which is 23.7%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1;php=zend
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - php=zend
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - php=zend
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - php=zend
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~scroll-down-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6001-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~payonlyproxy-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile8-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy4-hrd) - php=zend
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxycgi1-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pyproxy4-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~guidesites-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pyproxy0-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython7-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~usethisproxy-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython2-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile2-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~front-pages-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~appcodes-hrd) - php=zend
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf605-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk9-hrd) - php=zend
 www.bing.com/bingbot.htmimage/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6002-hrd) - php=zend
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 code.google.com/p/crawler4j/text/..crawler4j (url) - php=zend
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - php=zend
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - php=zend
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - php=zend
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaxminds) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jayjudah1) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - php=zend
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: jayjudah1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: carlos-web-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - php=zend
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - php=zend
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - php=zend
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - php=zend
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: img-proxy) - php=zend
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - php=zend
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - php=zend
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ownsterserver) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jakestokes) - php=zend
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: its-a-tr) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~fernandolsa-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: clh-sw) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sonydonin) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mahshighschool) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thepeterdeutsch) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khsprox) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - php=zend
 code.google.com/p/crawler4j/text/..crawler4j (url) -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: unblock4myspace) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - php=zend
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usingomega) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - php=zend
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: islom-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kushgenius) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zoxy-1) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ajprxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - php=zend
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: techjawaproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azzaziprxy) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaxminds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jerproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~iunblockedit) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: madkaiser2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ungoodthinkful) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kaveriselvaraj) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: goeverysite) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: student-pupil-help) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aboytes13tls) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - php=zend
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~twofrostyfoyou) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-web-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: encalada-proxy-one) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy) - php=zend
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: iprithi) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demowaiy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bakapouetprox) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zmomyanmar) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kakuro69) - php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hoptheborder) - php=zend
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/27.0.1453.110 Safari/537.36 AppEngine-Google; (url; appid: s~livescorefeed-hrd) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: prexyproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tinimarts) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~argentina-country) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: default-testing) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ancient-tractor-683) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - php=zend
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: batamsearch) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-data) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nuraiitj) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pramod-ajgaonkar-server) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yuricamara) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - php=zend
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sl-santabanta) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dr-use-me) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: izzy-queenofthefae) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujean) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yblk-ext-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thegatetoweb) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline0) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vibhoreindia) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: predictionwrong) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crowdsurfer100) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: myproxywx) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - php=zend
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lavprox) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: red-arg) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go6internet) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kittehserver) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~amort141) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jayjudah1) - -
 code.google.com/apis/kmltext/..Kml-Google; (url), gzip - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mistakeproxyarea) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy0) - php=zend
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~harunakaze) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: at2henning) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bie99miracle) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-eye-ear) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imabouncytigger) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boweiprox) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: caineserver) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nation4india) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mboharsik) - php=zend
 code.google.com/apis/kmlimage/..Kml-Google; (url), gzip - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - php=zend
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: funnyinternetwebsite) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz) - php=zend
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - php=zend
 0.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
 www.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; Facebookbot/2.1; url) en-US,en;q=0.5 php=zend
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 www.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
 0.facebook.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-us,zh;q=0.9 https=1;php=zend
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US, en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-us,zh;q=0.9 https=1
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US, en;q=0.5 php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn php=zend
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 php=zend
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/oggMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8,bn;q=0.6 php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - zero=470-01;php=zend
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 php=zend
 www.google.com/bot.htmltext/..mozzila/5.0 (compatible;GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
 www.google.com/bot.htmltext/..54.85.19.120 Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url ) en-US,en;q=0.5 php=zend
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US, en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - php=zend
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1;php=zend
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 php=zend
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - php=zend
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US php=zend
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US php=zend
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmlapplication/rsd+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai10) en-US php=zend
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~ydachao-9) en-US php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.2 - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai04) en-US php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai15) en-US php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro006) en-US php=zend
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - php=zend
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - php=zend
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - php=zend
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* php=zend
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1;php=zend
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - php=zend
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* php=zend
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) tr, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1;php=zend
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url; libwww-perl/5.814; SYSTRAN) - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01
 yandex.com/botstext/..198.143.33.65 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..198.143.47.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU -
 yandex.com/botstext/..198.143.47.65 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - php=zend
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
msn
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - https=1;php=zend
 search.msn.com/msnbot.htmtext/..(url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) -
cibra
 cibra.de/text/..CiBra Data Collector (url) - php=zend
 cibra.de/text/..CiBra Data Collector (url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou inst spider/4.0(url) zh-cn php=zend
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..web spider/4.0(url) - php=zend
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - php=zend
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - php=zend
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - php=zend
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - php=zend
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=zend
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=zend
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - php=zend
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
commoncrawl
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - php=zend
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stuckkey url pt-BR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/MusikAnimal url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Donner60 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Marcos_dias_de_oliveira url pt-BR,en,* php=zend
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBotapplication/jsonOrlodrimBot/1.0 (url) - php=zend
 ko.wikipedia.orgapplication/jsonurl - php=zend
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/wtmitchell url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.24.0 url - php=zend
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Startupevo1 url fr-FR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/I_dream_of_horses url en-US,* php=zend
 zero.wikipedia.orgtext/..Client Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/PhilKnight url en-GB,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Altsprachenfreund url de-DE,en,* php=zend
 zero.wikipedia.orgimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/KylieTastic url en-GB,* php=zend
 zero.wikipedia.orgtext/..IP Client Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/OWAIS_NAEEM url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.23.0 url - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/super48paul url nl-NL,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Vitor_Mazuco url pt-BR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Wikipelli url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/TheEpTic url en-GB,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Bollyjeff url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Lugia2453 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/John_Cline url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Petrb url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/9shi url zh-HK,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Cramunhao url de-DE,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Tutelary url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stickee url en-AU,* php=zend
wikiwand
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - php=zend
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
quickiwiki
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
 www.text/..Mozilla/5.0 (compatible; GoogleBot/2.1 urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) -
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.18.1 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.22.2 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-32-g0fd119d (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-193-g7be3479 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-124-gc2497e6 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.24.4 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.25.0-72-gd7d9f2f (url) en php=zend
 scrapy.orgtext/..Scrapy/0.22.0 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.24.2 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.16.4 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-32-g0fd119d (url) en -
 scrapy.orgtext/..Scrapy/0.24.4 (url) en https=1;php=zend
naver
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
 help.naver.com/robots/image/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.1 (Naver Corp.; url) TinEye/1.0 (via http://www.tineye.com/) - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - php=zend
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - php=zend
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - php=zend
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - php=zend
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - php=zend
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - php=zend
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - php=zend
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - php=zend
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - php=zend
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.19 - -
crossref
 det.labs.crossref.orgapplication/jsonCrossRef DET 3.4.7 - url -
zeerch
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - php=zend
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - php=zend
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - -
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - -
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - https=1;php=zend
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - https=1;php=zend
sblog
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs php=zend
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) cs php=zend
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) * php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) - php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/text/..(compatible; SeznamBot/3.2; url) - php=zend
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) - -
cloudflare
 www.cloudflare.com/always-onlineimage/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
 www.cloudflare.com/always-onlinetext/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - php=zend
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - php=zend
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - php=zend
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.12.13-gentoo; java 1.7.0_55; GMT09:00/ja) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; US/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p9; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-32-generic; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.5.0-44-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.16.1-1-ARCH; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.13-gentoo; java 1.7.0_55; GMT09:00/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.15.7-gentoo; java 1.7.0_55; Europe/ru) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.16.2-gentoo; java 1.7.0_65; GMT/ja) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_07; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.0-14-generic; java 1.8.0_20; Europe/ru) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/fr) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.13-gentoo; java 1.7.0_55; GMT09:00/ja) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/nl) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.41-042stab092.1; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmlimage/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/en) url en-us,en;q=0.5 php=zend
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - php=zend
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - php=zend
 tools.wmflabs.org/kmlexporttext/..url libwww-perl/6.03 - php=zend
 tools.wmflabs.org/geocommons/text/..url - php=zend
pagefreezer
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 php=zend
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 https=1;php=zend
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 php=zend
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - php=zend
plos
 alm.plos.orgapplication/jsonLagotto 3.5 - url - php=zend
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.4.8 - url - php=zend
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.4.8 - url - -
 sfo-stage-alm01.int.plos.orgapplication/jsonArticle-Level Metrics 3.4.8 - url - php=zend
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 php=zend
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - php=zend
wordpress
 goldendragon1971.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 halokue.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 hoangkimlong.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 drstvmovies.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 zeitgeist77.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 curtisnarimatsu.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 wildanrenaldi.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 hopefulandfree.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
etao
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - php=zend
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - php=zend
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 php=zend
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ar php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ru php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us
example
 www.example.com/text/..wikidrain/1.0 (url) - php=zend
 example.orgapplication/jsonArticle-Level Metrics 3.3.14 - url - php=zend
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - php=zend
 example.com/MyCoolToolPage/text/..User-Agent: MyCoolTool (url) - php=zend
 www.example.com/text/..wikidrain/1.0 (url) - -
unifiedlayer
 50-87-205-232.unifiedlayer.com/MadisonFindtext/..WordPress/4.0; url - php=zend
 173-254-124-63.unifiedlayer.com/LillianBesttext/..WordPress/4.0; url - php=zend
 50-87-201-249.unifiedlayer.com/HarperBesttext/..WordPress/4.0; url - php=zend
 50-87-0-170.unifiedlayer.com/ElianaBuy/love4buytext/..WordPress/4.0; url - php=zend
 173-254-123-200.unifiedlayer.com/SarahGreattext/..WordPress/4.0; url - php=zend
 50-87-201-32.unifiedlayer.com/AdrianaFasttext/..WordPress/4.0; url - php=zend
 173-254-103-226.unifiedlayer.com/KennediGaintext/..WordPress/4.0; url - php=zend
 173-254-99-209.unifiedlayer.com/MackenzieGurutext/..WordPress/4.0; url - php=zend
 50-87-201-102.unifiedlayer.com/NicoleSolutiontext/..WordPress/4.0; url - php=zend
 50-87-0-170.unifiedlayer.com/RebeccaGuidetext/..WordPress/4.0; url - php=zend
 50-87-203-21.unifiedlayer.com/KaitlynAcheivetext/..WordPress/4.0; url - php=zend
 50-87-201-32.unifiedlayer.com/IzabellaRecover/bestpowerhandtooltext/..WordPress/4.0; url - php=zend
 173-254-87-34.unifiedlayer.com/ArianaOnlinetext/..WordPress/4.0; url - php=zend
 50-87-2-24.unifiedlayer.com/CrystalCheap/greathandtoolstext/..WordPress/4.0; url - php=zend
 50-87-201-249.unifiedlayer.com/KatherineBig/babyonlinestufftext/..WordPress/4.0; url - php=zend
 50-87-35-135.unifiedlayer.com/HarleyValuetext/..WordPress/4.0; url - php=zend
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - php=zend
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - php=zend
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - php=zend
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - php=zend
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - php=zend
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
xovibot
 www.xovibot.net/text/..Mozilla/5.0 (compatible; XoviBot/2.0; url) - php=zend
 www.xovibot.net/text/..Mozilla/5.0 (compatible; XoviBot/2.0; url) - -
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - php=zend
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - -
wesee
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PageBot (url) - php=zend
 www.wesee.com/bot/text/..WeSEE:Ads/PictureBot (url) - -
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) - -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 php=zend
 www.diffbot.comimage/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - php=zend
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; Please Name Your robot; url) - php=zend
http:
 http://alm.example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - php=zend
xovi
 www.xovi.de/text/..Mozilla/5.0 (compatible; XoviOnpageCrawler; url) - php=zend
xbmc
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.3;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.0.36 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.0.50 armv7l; url) - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en php=zend
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - php=zend
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
riddler
 riddler.io/about.htmltext/..Riddler (url) - php=zend
 riddler.io/about.htmltext/..Riddler (url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - php=zend
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - php=zend
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - php=zend
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.htmltext/..Feedly/1.0 (url; like FeedFetcher-Google) - php=zend
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 php=zend
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
github
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/2.1.2p95 (url) en-us,en;q=0.5 php=zend
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 php=zend
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - php=zend
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/2.1.2p95 (url) en-us,en;q=0.5 -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - php=zend
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/2.0.0p481 (url) en-us,en;q=0.5 php=zend
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - php=zend
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * php=zend
uplabs0
 alm-1.uplabs0.comapplication/jsonArticle-Level Metrics 3.3.8 - url -
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de-DE, de;q=0.9 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) pr-BR, pt;q=0.8 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de-CH, fr-CH, it-CH, rm-CH, de;q=0.8, fr;q=0.8, it;q=0.8, rm;q=0.8 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de-AT, de;q=0.9 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) ro-RO, ro;q=0.9 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) hu-HU, hu;q=0.9 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) el-GR, el;q=0.9 php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de php=zend
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) th-TH, th;q=0.9 php=zend
aaronsw
 www.aaronsw.com/2002/feedfinder/text/..feedfinder/1.371 Python-urllib/1.16 url - php=zend
 www.aaronsw.com/2002/feedfinder/text/..feedfinder/1.371 Python-urllib/1.16 url - -
crazywebcrawler
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.0, url - php=zend
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.0, url - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - php=zend
 www.okian.ro/application/jsonMyBot/1.0 (url) - php=zend
drupal
 drupal.org/text/..Drupal (url) - php=zend
 drupal.org/application/jsonDrupal (url) - php=zend
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..Drupal (url) -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - php=zend
 www.saciol.com/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - php=zend
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
sputnik
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 php=zend
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 -
bluehost
 173-254-103-226.bluehost.com/LucilleUnlocktext/..WordPress/4.0; url - php=zend
 173-254-123-200.bluehost.com/DaniellaDirecttext/..WordPress/4.0; url - php=zend
 173-254-99-209.bluehost.com/CamrynUnlocktext/..WordPress/4.0; url - php=zend
 173-254-87-34.bluehost.com/PhoebeFasttext/..WordPress/4.0; url - php=zend
 173-254-122-134.bluehost.com/EmiliaFasttext/..WordPress/4.0; url - php=zend
 173-254-122-134.bluehost.com/GiulianaEasy/besthandtooltext/..WordPress/4.0; url - php=zend
 173-254-124-63.bluehost.com/IvyDirecttext/..WordPress/4.0; url - php=zend
 173-254-99-209.bluehost.com/IrisValuetext/..WordPress/4.0; url - php=zend
leiki
 www.leiki.comtext/..Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; url) - php=zend
 www.leiki.comtext/..Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; url) - -
mysite
 www.mysite.com/text/..MyBot/1.0 (url) - php=zend
 www.mysite.com/application/vnd.php.serializedMyBot/1.0 (url) - php=zend
 www.mysite.com/application/jsonMyBot/1.0 (url) - php=zend
 www.mysite.com/botinfoapplication/vnd.php.serializedMozilla/5.0 (compatible; MyCoolBot/1.0; url) - php=zend
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - php=zend
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - php=zend
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=zend
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - php=zend
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.2.0; url ) en-us,en;q=0.5 php=zend
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) en-us,en;q=0.5 php=zend
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) en-us,en;q=0.5 -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.2.0; url ) - php=zend
kinshoo
 www.kinshoo.com/bot.htmltext/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 php=zend
 www.kinshoo.com/bot.htmlimage/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 -
draigweb
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - php=zend
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - php=zend
datasift
 datasift.com/bot.htmltext/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - php=zend
 datasift.com/bot.htmlimage/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - -
toolserver
 toolserver.org/~dispenser/application/jsonDispensersTools (url) - php=zend
 toolserver.org/~dispenser/text/..DispensersTools (url) - php=zend
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - php=zend
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - php=zend
test
 test.co.kr/text/..xxxxxxxx/Nutch-2.2.1 (url) - php=zend
inaf
 marlin.iasfbo.inaf.it/ctatext/..WordPress/3.9.1; url - php=zend
geolocation
 www.geolocation.wsapplication/jsonGeolocation.ws import. url -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 php=zend
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
site
 www.site.com/application/jsonBot/2.0 (url) -
gorgor
 gorgor.irtext/..hivaBot/hivaBot-1.0 (Iranian Search Engine bot; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 gorgor.irtext/..hivaBot/hivaBot-1.0 (Iranian Search Engine bot; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - php=zend
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
crowdometer
 labs.crowdometer.orgapplication/jsonLagotto 3.6-beta - url - php=zend
pchome
 www.pchome.com.tw/pchomebot.htmtext/..Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 php=zend
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * php=zend
wikidict
 www.wikidict.detext/..url - php=zend
qbqm
 www.qbqm.fr/text/..QBQM/1.0 (url) fr php=zend
netvibes
 www.netvibes.comtext/..Netvibes (url) - php=zend
 www.netvibes.comapplication/jsonNetvibes (url) en-us php=zend
 www.netvibes.comapplication/xmlNetvibes (url) - php=zend
verizon
 www.verizon.comapplication/jsonCrawler/1.0 (url; mail address ) Custom/1.0 - php=zend
 www.verizon.comapplication/jsonCrawler/1.0 (url; mail address ) Custom/1.0 - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - php=zend
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
SearchNearMe
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - php=zend
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - php=zend
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - php=zend
pinterest
 www.pinterest.com/text/..Pinterest/0.2 (url) - php=zend
 www.pinterest.com/image/..Pinterest/0.2 (url) - -
 pinterest.com/image/..Pinterest/0.1 url - -
howmuchdohomesecuritysystemscost
 warnerrobins.howmuchdohomesecuritysystemscost.comtext/..WordPress/3.9.2; url - php=zend
 homesecurity.howmuchdohomesecuritysystemscost.comtext/..WordPress/3.9.2; url - php=zend
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - php=zend
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=zend
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - php=zend
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - php=zend
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - php=zend
edu
 www.aau.edu.ettext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url for Ethiopian Web archive project) - php=zend
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - php=zend
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - php=zend
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - php=zend
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - php=zend
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - php=zend
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - php=zend
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - php=zend
soccerly
 womens.soccerly.com/cgi-bintext/..WordPress/3.9.2; url - php=zend
ijs
 ailab.ijs.siapplication/xmlNewsMachine/0.1 (url; mail address ) BasedOnJava/1.7 - php=zend
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja php=zend
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - php=zend
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - php=zend
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - php=zend
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - php=zend
servicesonrequest
 www.servicesonrequest.comtext/..User-Agent: WikiBot (url) - php=zend
 www.servicesonrequest.com/application/jsonWikiBot(url) - php=zend
perfectmarket
 www.perfectmarket.comtext/..Mozilla/5.0 (compatible; PerfectMarketKWTbot/1.0 url) - php=zend
 www.perfectmarket.comtext/..Mozilla/5.0 (compatible; PerfectMarketKWTbot/1.0 url) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1 (url) en php=zend
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en php=zend
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) it php=zend
ac
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - php=zend
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - php=zend
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - php=zend
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - php=zend
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - php=zend
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - php=zend
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.13 (url) - php=zend
xxx
 www.xxx.comtext/..spider/Nutch-1.5.1 (spider; url) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
simplepie
 simplepie.orgtext/..SimplePie/1.3.1 (Feed Parser; url; Allow like Gecko) Build/20121030175403 - php=zend
 simplepie.orgapplication/xmlSimplePie/1.3.1 (Feed Parser; url; Allow like Gecko) Build/20121030175403 - php=zend
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
metamagazine
 metamagazine.comtext/..WordPress/3.9.2; url -
profound
 www.profound.net/domainappendertext/..Mozilla/5.0 (compatible; DomainAppender /1.0; url) - php=zend
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - php=zend
addsearch
 www.addsearch.com/bot/image/..Mozilla/5.0 (compatible; AddSearchBot; url) en-US,* -
 www.addsearch.com/bot/text/..Mozilla/5.0 (compatible; AddSearchBot; url) - php=zend
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - php=zend
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) -
webthumbnail
 webthumbnail.orgimage/..Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; url) en,* -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - php=zend
 www.picsearch.com/bot.htmltext/..psbot-image (url) - php=zend
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=zend
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawlerimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 php=zend
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 php=zend
idmarch
 www.idmarch.org/bot.htmltext/..Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; url) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - php=zend
grouphigh
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - php=zend
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - php=zend
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - php=zend
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - php=zend
inagist
 inagist.comtext/..InAGist URL Resolver (url) en-us,en;q=0.5 php=zend
localhost
 localhost/clumpit/bot.phptext/..Mozilla/5.0 (compatible; clumboot; url) - php=zend
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - php=zend
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.3 (url) - php=zend
wikidata
 www.wikidata.orgapplication/jsonurl - php=zend
bestonlinesongs
 www.bestonlinesongs.com/text/..MyBot/1.0 (url) - php=zend
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - php=zend
 www.thinglink.com/help/ThinglinkImageBottext/..Elmer, the Thinglink ImageBot (url) - php=zend
tineye
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - php=zend
blogspot
 alberapps.blogspot.comtext/..TiempoBus/3.1.4 (url; mail address ) - php=zend
 alberapps.blogspot.comtext/..TiempoBus/2.8 (url; mail address ) - php=zend
QtWeb
wikilovesearth
 wikilovesearth.at/tools/text/..wle_at/1.0 (url; mail address ) - php=zend
site-shot
 www.site-shot.com/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Chromium/31.0.1650.63 Site-Shot/2.1 (url) Safari/534.34 en-US,* -
 www.site-shot.com/text/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Chromium/31.0.1650.63 Site-Shot/2.1 (url) Safari/534.34 en-US,* php=zend
innercitylist
 classified.innercitylist.comtext/..WordPress/4.0; url - php=zend
 atlanta.innercitylist.comtext/..WordPress/4.0; url - php=zend
sprawk
 www.sprawk.com/text/..Sprawk/1.4 (url) - php=zend
scarecrow
 www.scarecrow.com/application/jsonScarecrow/1.0 (url; mail address ) - php=zend
temnos
 temnos.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 php=zend
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) - php=zend
datoid
 datoid.cz/tag-information/application/jsonDatoid/0.9 (url; mail address ) BasedOnNette/2.0.9 - php=zend
livelap
 site.livelap.com/crawlertext/..LivelapBot/0.2 (url) en-US,en php=zend
 site.livelap.com/crawlerimage/..LivelapBot/0.2 (url) en-US,en -
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - php=zend
creativepulses
 creativepulses.nltext/..CreativePulses Crawler (url) - php=zend
themovementcenterforyoga
 themovementcenterforyoga.comtext/..WordPress/3.9.2; url - php=zend
205711.09999998total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - php=zend
 application/vnd.php.serialized
 text/..
 application/json
GoogleBot-Image/1.0 - -
 image/..
 text/..
php wikibot classes - php=zend
 application/vnd.php.serialized
 text/..
MediaWikiCrawler-Google/2.0 ( mail address ) - php=zend
 text/..
pwb/rg3428 Pywikibot/2.0 - php=zend
 application/json
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN php=zend
 text/..
 application/json
 application/xml
ClueBot/2.0 - php=zend
 application/vnd.php.serialized
 text/..
LinkParser/2.0 - php=zend
 text/..
GoogleBot-Image/1.0 - php=zend
 text/..
 image/..
wikiwix-bot-3.0 - php=zend
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
ProoXiBot fr php=zend
 text/..
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
py-radehh.py/rg10975 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
maj_articles_recents/r3778 Pywikipediabot/2.0 - php=zend
 application/json
 text/..
milog_bot/1.0 ( mail address ) - php=zend
 text/..
tigerbot - php=zend
 application/json
 text/..
pywikibot-compat-redirect.py/rg10993 Pywikipediabot/1.0 - php=zend
 application/json
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
pwb/rg3428 Pywikibot/2.0 - -
 application/json
 text/..
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 php=zend
 text/..
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - php=zend
 application/json
 text/..
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - -
 application/vnd.php.serialized
 text/..
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
MediaWiki::Bot/5.005006 - php=zend
 application/json
 text/..
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; php=zend
 text/..
pywikipedia-radeh.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
MediaWiki::Bot/3.2.6 - php=zend
 application/json
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
 application/xml
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - php=zend
 application/json
AniBot/0.9 php/curl - php=zend
 application/vnd.php.serialized
 text/..
Twitterbot/1.0 - php=zend
 text/..
DigitalsmithsBot - php=zend
 text/..
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
RBot/0.3 ( mail address ) -
 application/json
 image/..
 text/..
PythonWikipediaBot/1.0 - php=zend
 application/json
 text/..
pywikibot-1newar25.py/rg11061 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
wymowa (commons:commons; User:Alkamid) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pwb/rg3113 Pywikipediabot/2.0 - php=zend
 application/json
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - php=zend
 text/..
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11060 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
GoogleBot-Image/1.0 - php=zend
 text/..
 application/json
 image/..
UAA-UAA.py/r10417 Pywikipediabot/1.0 - php=zend
 application/json
web-crawler en-US,en;q=0.8 php=zend
 text/..
Screaming Frog SEO Spider/2.55 - php=zend
 text/..
Google Bot - php=zend
 application/json
 text/..
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Facebot/1.0 -
 text/..
 application/ogg
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - php=zend
 application/json
Owlin Bot v. 3.0 - php=zend
 text/..
php wikibot classes - -
 application/vnd.php.serialized
 text/..
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
ClueBot/2.0 - -
 application/vnd.php.serialized
pywikipedia-imageuncat.py/rg10972 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
MediaWiki::Bot/3.005002 - php=zend
 application/json
 text/..
SecretaryBot (https://github/vlsergey/Secretary; mail address ) BasedOnJWBF - php=zend
 text/..
 application/json
ProoXiBot fr -
 text/..
 video/webm
 application/ogg
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN -
 text/..
 image/..
LinkParser/2.0 - -
 text/..
pywikipedia-git-radesaz.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
wikiwix-bot-3.0 - -
 text/..
Arachnida Web Crawler - php=zend
 text/..
python-wikitools/1.2 (User:DickensBot) -
 application/json
 text/..
GoogleBot/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
pywikipedia-newar.py/rg11037 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* php=zend
 text/..
 image/..
 application/json
GoogleBot-Image/1.0 - https=1;php=zend
 text/..
YisouSpider - php=zend
 text/..
 application/json
iw2data/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
 text/..
pywikipedia-newinterwiki-fa.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
 text/..
addIdentifierMappingsProteinBoxBot/rg3599 Pywikibot/2.0 - php=zend
 application/json
 text/..
WikidataBot framework - php=zend
 application/json
pywikibot-1newar25.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-git-wdlabel.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
py-interwiki.py/rg11047 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - php=zend
 application/json
NutchSpider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
py-radehh.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
Yeobot/1.1 Contact: mail address ) - php=zend
 text/..
Twitterbot/1.0 - -
 image/..
 text/..
 application/pdf
 application/ogg
 video/webm
 audio/webm
BeneBot*/1.0 WikibasePhpLib/0.1 - php=zend
 application/json
Egerin Spider 1.0 - php=zend
 text/..
pywikipedia-radeh.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-git-fa_bot.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
MediaWiki::Bot/5.005007 - php=zend
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
 application/xml
ContextAd Bot 1.0 - php=zend
 text/..
spider - php=zend
 text/..
 application/json
 application/vnd.php.serialized
SPIupdater-globalfunc.py/r10417 Pywikipediabot/1.0 - php=zend
 application/json
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
compat-featured.py/r10441 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
compat-checkimages.py/rs11029 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlBot) - php=zend
 text/..
GermCrawler - php=zend
 application/json
 text/..
 application/xml
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - php=zend
 application/json
 text/..
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - php=zend
 application/json
BotReversor-BotReversor.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
MyCuteBot/0.1 - php=zend
 text/..
 application/json
pywikipedia-zzredirect.py/r10582 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
SineBot/1.5.19(User:SineBot) - php=zend
 application/vnd.php.serialized
 text/..
SocialBot - php=zend
 application/json
cwikt-interwiki.py/rg11060 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - php=zend
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - php=zend
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
CorenSearchBot/1.7 en libwww-perl/6.03 - php=zend
 text/..
AnomieBOT/1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - php=zend
 application/json
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - php=zend
 text/..
refreshfeeds (wikinews:ru; User:AKBot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
ifd-empty (wikidata:wikidata; User:PLbot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
EntrezGeneStubBot/rg3599 Pywikibot/2.0 - php=zend
 application/json
pywikipedia-zzinter.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
 text/..
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - php=zend
 text/..
 application/json
DotNetWikiBot/3.0 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - php=zend
 text/..
User-Agent Baiduspider - php=zend
 text/..
 image/..
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - php=zend
 text/..
 application/xml
pwb/rg3113 Pywikipediabot/2.0 - -
 application/json
compat-basictemplatedisamblist.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikibot-compat-commonscat.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
maj_articles_recents/r3778 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzfatranslator.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
WorldOfMusicBetaBot/1.0 - php=zend
 text/..
AnomieBOT/1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - php=zend
 application/json
KatBot bot.nytlabs.com> - php=zend
 text/..
replace (wiktionary:it; User:Barbabot) Pywikibot/2.0b1 (s5161) httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
compat-basictemplatedisamblist.py/rg11060 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 php=zend
 text/..
 -
 application/json
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - php=zend
 application/json
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
SocialBot - -
 application/json
pywikibot-compat-commonscat.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Facebot/1.0 - php=zend
 text/..
 application/json
bot-07-fix-language-mappings (wikidata:wikidata; User:AkkakkBot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - php=zend
 application/json
HTMLParser/2.0 - php=zend
 text/..
BOT/0.1 (BOT for JCE) - php=zend
 text/..
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-aaa_categorieAutori.py/rg10952 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
tigerbot - -
 application/json
 text/..
 image/..
 application/ogg
Java Bot (https://github/vlsergey/Secretary; mail address ) BasedOnJWBF - php=zend
 application/json
 text/..
Nutch Spider/Nutch-2.2.1 - php=zend
 text/..
pywikipedia-git-sectionBoten.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
ko_missing/-1 (unknown) Pywikibot/2.0 (User:Chobot) - php=zend
 application/json
AnomieBOT/1.0 (TemplateSubster; see [[User:AnomieBOT]]) - php=zend
 application/json
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/rg11061 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzredirectyeh.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
milog_bot/1.0 ( mail address ) - -
 text/..
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
GoogleBot/GoogleBot 1.0 (GoogleBot; www.google.com; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
GoogleBot (gocrawl v0.4) - php=zend
 text/..
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - php=zend
 application/vnd.php.serialized
Webwiki Search Engine Bot - www.webwiki.de - php=zend
 text/..
wymowa (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
SentiBot www.sentibot.eu (compatible with GoogleBot) - php=zend
 text/..
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - php=zend
 text/..
GjirafaSpider - php=zend
 text/..
Webbot - php=zend
 text/..
PythonWikipediaBot/1.0 -
 application/json
 text/..
Goalkeeperbot(User:Beetstra)/1.0 - php=zend
 text/..
pwb/rg3045 Pywikipediabot/2.0 - php=zend
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 php=zend
 text/..
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.60; ) - php=zend
 text/..
 application/xml
pywikibot-compat-commonscat.py/rg11060 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Wikibot/2.1.1 CFNetwork/672.1.15 Darwin/14.0.0 en-us -
 image/..
 text/..
XLinkBot/1.00 - php=zend
 text/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-featured.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
MakeScoreTable/s4944 Pywikibot/2.0 (User:) - php=zend
 application/json
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 text/..
MakeScoreTable-alter/s4944 Pywikibot/2.0 (User:) - php=zend
 application/json
 text/..
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - php=zend
 text/..
 application/xml
SurakWare MediaWiki Bot/1.0 - php=zend
 text/..
 application/xml
MediaWiki::Bot/3.2.6 - -
 application/json
pywikibot-compat-aaa_categorieTesti.py/rg10952 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
mesDemanats4/g3814 Pywikibot/2.0 (User:CastorBot) - php=zend
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1;) en-US,en;q=0.5 php=zend
 text/..
pywikibot-1newar25.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-git-maghalesaz_new.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11057 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - php=zend
 application/json
pwb/rg3369 Pywikibot/2.0 - php=zend
 application/json
compat-label_bot.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
compat-basicarticledisamblist.py/rg11060 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-radeh.py/rg11050 Pywikipediabot/1.0 - -
 application/json
pwb/rg3536 Pywikibot/2.0 - php=zend
 application/json
MediaWiki::Bot/5.005006 - -
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - php=zend
 text/..
pywikipedia-git-maghalesaz_cmd2.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 (compatible; SputnikImageBot/2.3) - -
 image/..
 text/..
articles/r2554 Pywikipediabot/2.0 - php=zend
 application/json
Planet OS Crawler/Nutch-1.9 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 application/pdf
t9/rg2816 Pywikipediabot/2.0 - php=zend
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 application/ogg
 image/..
pywikibot-compat-redirect.py/rg10993 Pywikipediabot/1.0 - -
 application/json
extract_dates/g3914 Pywikibot/2.0 (User:BotMultichill) - php=zend
 application/json
pywiki-geowiki.py/rg11061 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-git-maghalesaz2.py/rg90 Pywikipediabot/1.0 - -
 application/json
ifd-hoobot (wikidata:wikidata; User:PLbot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-zzfatranslator.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 application/json
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - php=zend
 text/..
pywikipedia-git-radesaz.py/rg90 Pywikipediabot/1.0 - -
 application/json
compat-desc_bot.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
compat-basicarticledisamblist.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-checkimages.py/rg10976 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-git-maghalesaz2.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11066 Pywikipediabot/1.0 - php=zend
 application/json
DigitalsmithsBot - -
 text/..
RokaSpider/Nutch-1.8 (' mail address ') de-de,de,en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us php=zend
 text/..
pywikipedia-newinterwiki-fa.py/rg11028 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
compat-checkimages.py/rs11045 Pywikipediabot/1.0 - php=zend
 application/json
erfgoedbot-categorize_images.py/rg10979 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
DotNetWikiBot/3.10 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
py-navbox.py/rg10975 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikibot-compat-cfd.py/rg11060 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
compat-redirect.py/rs11026 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
redirect/s4970 Pywikibot/2.0 (User:Xqbot) - php=zend
 application/json
4210987 (wikipedia:en; User:RussBot) Pywikibot/2.0b1 (g4158) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - php=zend
 application/json
Peachy MediaWiki Bot API Version 1.0 - php=zend
 application/vnd.php.serialized
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US php=zend
 text/..
 image/..
compat-redirect.py/rs11044 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
pwb/r3152 Pywikipediabot/2.0 - php=zend
 application/json
pywiki-lit.py/rg11065 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
pywiki-welcome.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
myvboBot/Nutch-1.8 (j x c x e AT z x i x f x t x r DOTCOM (remove 'x' chars)) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
 application/opensearchdescription+xml
DotNetWikiBot/2.92 (Microsoft Windows NT 6.1.7600.0; ) - php=zend
 text/..
 application/xml
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-zzaddnavebox.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-zzfatranslator.py/rg11047 Pywikipediabot/1.0 - php=zend
 application/json
AnomieBOT/1.0 (BAGBot; see [[User:AnomieBOT]]) - php=zend
 application/json
 text/..
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
AnomieBOT/1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - php=zend
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X; en-us) AppleWebKit/534 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 MsnBot-Media /1.0b - php=zend
 text/..
 -
 application/json
mijnbots-NavSjabOpnemen.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11050 Pywikipediabot/1.0 - -
 application/json
cwikt-interwiki.py/rg11050 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikibot-compat-sichteroptin.py/r10883 Pywikipediabot/1.0 - php=zend
 text/..
 application/json
cwikt-interwiki.py/rg11057 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
DotiiBot - php=zend
 text/..
addEntrezGeneInBatch/rg3599 Pywikibot/2.0 - php=zend
 application/json
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
 text/..
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-radeh.py/rg11050 Pywikipediabot/1.0 - php=zend
 application/json
DeclensionParser/0.02 ( mail address ) - php=zend
 text/..
interwiki/g3936 Pywikibot/2.0 (User:) - php=zend
 application/json
mussol-mussol.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11067 Pywikipediabot/1.0 - php=zend
 application/json
My Nutch Spider/Nutch-1.9 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
pywikipedia-zzgallery.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - -
 application/json
 text/..
gyffu_bot - php=zend
 text/..
py-importwdn2.py/rs11047 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
Crawler/1.2 - php=zend
 text/..
pywikilab-llista_mils.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1; en-US,en;q=0.5 php=zend
 text/..
vm-auto-erl-neu/s4959 Pywikibot/2.0 (User:Xqbot) - php=zend
 application/json
node.js - nodemw - mail address - Maintainer: Rainer Rillke - mail address - [[:commons:User:Rillke]] - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
My Nutch Spider (www.yurichev.com; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
compat-checkimages.py/rs11044 Pywikipediabot/1.0 - php=zend
 application/json
pywiki-geowiki.py/rg11065 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-zzpurging.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
German Wikipedia Broken Weblinks Bot; contact: mail address - php=zend
 text/..
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
node-crawler/0.2.5 - php=zend
 text/..
pywikipedia-commonscat.py/rg11057 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - php=zend
 application/xml
 text/..
PHPCrawl - php=zend
 text/..
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - php=zend
 application/json
coordinate_import/g3916 Pywikibot/2.0 (User:AyackBot) - php=zend
 application/json
pywikibot-1newar25.py/rg11057 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - php=zend
 text/..
pwb/rg3012 Pywikipediabot/2.0 - php=zend
 application/json
HTMLParser/1.6 - php=zend
 text/..
moogolbot/Moogol-1.0 (moogolbot) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
ntkbot/Nutch-1.8 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
PythonWikipediaBot/1.0 - -
 application/json
 text/..
UKBot [[:no:Bruker:UKBot]] - MwClient/0.7.0dev (https://github.com/mwclient/mwclient) - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 application/json
pywikipedia-zzfatranslator.py/rg11047 Pywikipediabot/1.0 - -
 application/json
python (wikidata:wikidata; User:Frettiebot) Pywikibot/2.0b1 httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
RSS Bot/1.9 (Mac OS X Version 10.9.4 (Build 13E28)) en;q=1, ja;q=0.9, fr;q=0.8, de;q=0.7, es;q=0.6, it;q=0.5 php=zend
 application/xml
 -
Mozilla/5.0 (compatible; MerchantCentricBot/1.0) en-US,* php=zend
 text/..
compat-welcome.py/rs11039 Pywikipediabot/1.0 - php=zend
 application/json
Screaming Frog SEO Spider/2,55 - php=zend
 text/..
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
pywikibot-1newar25.py/rg11050 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 zh-CN,zh;q=0.8,en;q=0.6 -
 image/..
 text/..
UAA-UAA.py/r10417 Pywikipediabot/1.0 - -
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
COIBot/2.0 - php=zend
 text/..
porzucone/g3936 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
pywikipedia-newar.py/rg11037 Pywikipediabot/1.0 - -
 application/json
 application/xml
run/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
pywikipedia-git-g_badge_killer.py/rg90 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT/1.0 (WikiProjectWorker; see [[User:AnomieBOT]]) - php=zend
 application/json
Xalted Crawlers/WebCrawler (Testing) en-us, en-gb, en;q=0.7, *;q=0.3 php=zend
 text/..
pywikipedia-git-albot.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
pywikilab-bot_viquirepte.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
My Bot - php=zend
 text/..
AnomieBOT/1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - php=zend
 application/json
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - php=hhvm
 application/json
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US php=zend
 -
 text/..
 image/..
 application/json
Owlin Bot v. 3.0 - -
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; ) - php=zend
 text/..
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
MoodleBot/1.0 - php=zend
 application/xml
 application/vnd.php.serialized
 text/..
pywikipedia-git-importwdn3.py/rg90 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-git-g_badge_killer.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
 text/..
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-wikifyer.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
archiv-erl-bot/rg2954 Pywikipediabot/2.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11050 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Screaming Frog SEO Spider/2.55 - -
 image/..
 text/..
 application/ogg
pywikipedia-git-badge_killer.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; fr-fr; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 fr-FR, en-US -
 image/..
 -
 text/..
pywikibot-1newar25.py/rg11050 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - php=zend
 text/..
 application/xml
 application/json
logpatrol (wikipedia:th; User:Nullzerobot) Pywikibot/2.0b1 (g4018) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11065 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-parserfunctioncount.py/r10809 Pywikipediabot/1.0 - php=zend
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
pwb/r3673 Pywikibot/2.0 - php=zend
 application/json
ClearBot/1.0 - php=zend
 text/..
pywikipedia-imageuncat.py/rg10972 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzredirectyeh.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US php=zend
 -
 text/..
 image/..
 application/json
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - php=hhvm
 application/json
iw_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
request-daemon/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
porzucone/g3949 Pywikibot/2.0 (User:AlkamidBot) - php=zend
 application/json
porzucone/g3919 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
HTMLParser/2.0 - -
 text/..
pywikipedia-interwiki.py/rg13 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
redirect (wikipedia:en; User:Xqbot) Pywikibot/2.0b1 (s5189) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
WikiBot/0.1 - php=zend
 text/..
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
redirect (wikipedia:nl; User:Xqbot) Pywikibot/2.0b1 (s5189) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11066 Pywikipediabot/1.0 - php=zend
 application/json
redirect (wikipedia:en; User:Xqbot) Pywikibot/2.0b1 (s5104) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-zzfawikipatroler.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
MWFeedParser de-de php=zend
 application/xml
pywikibot-compat-commonscat.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11067 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/4.0 (compatible; ISYS Web Spider 9) - php=zend
 text/..
pywikipedia3-welcome.py/r11747 Pywikipediabot/1.0 - php=zend
 application/json
wymowa/g3954 Pywikibot/2.0 (User:Alkamid) - php=zend
 application/json
Facebot/1.0 - https=1;php=zend
 text/..
robot leapit.com en-US,* -
 image/..
 text/..
pywikipedia-git-featured_touch.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
 application/xml
YOUBot (Youniversal Media Bot) - -
 image/..
 text/..
Baidu-YunGuanCe-Bot(ce.baidu.com) - php=zend
 text/..
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
python-wikitools/1.2 (User:Mr.Z-bot) - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.3; ru-ru; Spider Build/JLS36C) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 ru-RU, en-US -
 image/..
 application/json
AnomieBOT/1.0 (NewArticleAFDTagger; see [[User:AnomieBOT]]) - php=zend
 application/json
cdh-delinker.py/r11775 Pywikipediabot/1.0 - php=zend
 application/json
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
 -
vm-auto-erl-neu (wikipedia:de; User:Xqbot) Pywikibot/2.0b1 (s5183) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
mgverbscript/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
claimit/g3936 Pywikibot/2.0 (User:Whymbot) - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
Botzilla-av_bot.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - -
 application/json
mijnbots-NavSjabOpnemen.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
Google Bot - -
 application/json
redirect (wikipedia:nl; User:Xqbot) Pywikibot/2.0b1 (s5104) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
py-interwiki.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - php=zend
 text/..
 application/xml
cd-delinker.py/r11775 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
MediaWiki::Bot/5.005007 - -
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
frettiebot-python.py/rg11064 Pywikipediabot/1.0 - php=zend
 application/json
wymowa/g3930 Pywikibot/2.0 (User:Alkamid) - -
 application/json
Full_update/r2554 Pywikipediabot/2.0 - php=zend
 application/json
CaBot - php=zend
 text/..
creator_link/g3914 Pywikibot/2.0 (User:BotMultichill) - php=zend
 application/json
delete_template (wikipedia:ru; User:WinterheartBot) Pywikibot/2.0b1 (g4171) httplib2/0.9 Python/2.7.7.final.0 - php=zend
 application/json
setup (wikidata:wikidata; User:JVbot) Pywikibot/2.0b1 (g4123) httplib2/0.9 Python/3.4.1.final.0 - php=zend
 application/json
pywikipedia-zzredirectyeh.py/rg11050 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
ContextAd Bot 1.0 - -
 text/..
 image/..
pywiki-zzCityLocation.py/rg11061 Pywikipediabot/1.0 - php=zend
 application/json
commonscats/r2554 Pywikipediabot/2.0 - php=zend
 application/json
pywikipedia-git-desc_bot2.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Peachy MediaWiki Bot API - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
pywikipedia-commonscat.py/rg11060 Pywikipediabot/1.0 - php=zend
 application/json
dumpprocessor/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
32069.02total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Dec 8, 2014 14:21
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers