Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Oct 2014 - 31 Oct 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 143,597,650 page requests (mime type text/html only!) per day are considered crawler requests, out of 668,924,970 external requests, which is 21.5%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1;php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - php=zend
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - php=zend
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~cgiproxy3-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6001-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~payonlyproxy-hrd) - php=zend
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile8-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy4-hrd) - php=zend
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxycgi1-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~guidesites-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy-py-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~lookingforwe-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~scroll-down-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~front-pages-hrd) - php=zend
 www.bing.com/bingbot.htmimage/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~overviewa-hrd) - php=zend
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pyproxy0-hrd) - php=zend
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 code.google.com/p/crawler4j/text/..crawler4j (url) - php=zend
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - php=zend
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - php=zend
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - php=zend
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~jnhanproxy) - php=zend
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaxminds) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jayjudah1) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: jayjudah1) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: carlos-web-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - php=zend
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - php=zend
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~argentina-country) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mahshighschool) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - php=zend
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - php=zend
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: clh-sw) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - php=zend
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaxminds) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khsprox) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~ax-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jakestokes) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - php=zend
 code.google.com/p/ldspider/wiki/Robotstext/..ldspider (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kushgenius) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - php=zend
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ownsterserver) - php=zend
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sonydonin) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ungoodthinkful) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - php=zend
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azzaziprxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: goeverysite) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ajprxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: student-pupil-help) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thepeterdeutsch) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: img-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aplitcomposition) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~projection11111) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~imgdaili3) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: encalada-proxy-one) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: its-a-tr) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: myproxywx) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: default-testing) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: predictionwrong) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~safebrowsing88) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bakapouetprox) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nicoflysurf) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-web-proxy) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~e-ddos) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - php=zend
 www.google.com/adsbot.htmltext/..AdsBot-Google (url) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aboytes13tls) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jerproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yes9987) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: opensite4me) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: izzy-queenofthefae) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: islom-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~no8lancelot) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - php=zend
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ffffnm) zh-CN,zh;q=0.8,zh-TW;q=0.6 php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: unblock4myspace) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: optusabuse) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - php=zend
 code.google.com/appengineapplication/jsonPython-urllib/2.7 AppEngine-Google; (url; appid: s~context-engine) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hoptheborder) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sl-santabanta) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~education-center) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zmomyanmar) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~twofrostyfoyou) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: techjawaproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ms-office) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: madkaiser2) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: downarchivestuffproxyserver) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demowaiy) - php=zend
 code.google.com/appenginetext/..Mozilla/3.0 (compatible) AppEngine-Google; (url; appid: s~bdisphacking5) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vibhoreindia) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nikita-web-proxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mapremiereapplication) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nuraiitj) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - php=zend
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/27.0.1453.110 Safari/537.36 AppEngine-Google; (url; appid: s~livescorefeed-hrd) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mistakeproxyarea) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~lime-free) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kevsproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bie99miracle) - php=zend
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: issamwap1) - php=zend
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~harunakaze) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~warawasetekure) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~iunblockedit) - php=zend
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - php=zend
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - php=zend
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-us,zh;q=0.9 https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1;php=hhvm
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US php=zend
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - https=1;php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bn,en-us;q=0.7,en;q=0.3 -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - https=1;php=zend
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,bn;q=0.7,en;q=0.3 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 php=zend
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmlapplication/oggMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 php=zend
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - php=zend
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1;php=zend
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozillah/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - php=zend
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)general.useragent.override en-US,en;q=0.5 php=zend
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 php=zend
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ru,en-us;q=0.7,en;q=0.7 php=zend
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 php=zend
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - php=zend
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - php=zend
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* php=zend
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - php=zend
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1;php=zend
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-US,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - php=zend
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-US,* php=zend
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* php=zend
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 php=zend
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - php=zend
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1;php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - php=zend
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en,* php=zend
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - php=zend
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spi_der.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - php=zend
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US php=zend
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw php=zend
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - php=zend
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - php=zend
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - php=zend
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - php=zend
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US https=1;php=zend
 www.baidu.com/search/spider.htmltext/..mozilla/5.0 (compatible; baiduspider/2.0; url)Accept: text/html,application/xhtmlxml,application/xml;image/png,image/jpeg,image/*;q=0.9,*/*;q=0.8 en-us;q=0.7, en;q=0.3 php=zend
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - php=hhvm
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url),gzip(gfe) - php=zend
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) tr, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01
 yandex.com/botstext/..198.143.47.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1;php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..198.143.33.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - php=zend
 yandex.com/botstext/..198.143.34.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botstext/..198.143.47.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsapplication/vnd.php.serializedMozilla/5.0 (compatible; YandexBot/3.0; url) ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..198.143.34.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - php=zend
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU -
 yandex.com/botstext/..198.143.47.33 Mozilla/5.0 (compatible; YandexBot/3.0; url) - php=zend
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us,en;q=0.5
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - php=zend
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - php=hhvm
cibra
 cibra.de/text/..CiBra Data Collector (url) - php=zend
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - php=zend
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
msn
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - php=zend
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - https=1;php=zend
commoncrawl
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - php=zend
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - php=zend
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - php=zend
 wikipedia.orange.fr/text/..API/1.0 (url) - php=hhvm
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - php=zend
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - php=zend
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn php=zend
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou inst spider/4.0(url) zh-cn php=zend
 www.sogou.com/docs/help/webmasters.htm#07text/..web spider/4.0(url) - php=zend
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=zend
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - php=zend
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 php=zend
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - php=zend
naver
 help.naver.com/robots/application/jsonMozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
 help.naver.com/robots/text/..Mozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
 help.naver.com/robots/image/..Mozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/application/jsonMozilla/5.0 (compatible; Yeti/1.1; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=hhvm
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 php=zend
quickiwiki
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - php=zend
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/MusikAnimal url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/I_dream_of_horses url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.24.0 url - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Donner60 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Marcos_dias_de_oliveira url pt-BR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/wtmitchell url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/HeicoH url de-DE,en,* php=zend
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBotapplication/jsonOrlodrimBot/1.0 (url) - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/PauloEduardo url pt-BR,en,* php=zend
 zero.wikipedia.orgtext/..Client Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Vitor_Mazuco url pt-BR,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Altsprachenfreund url de-DE,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/KylieTastic url en-GB,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.23.0 url - php=zend
 ko.wikipedia.orgapplication/jsonurl - php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/John_Cline url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Lugia2453 url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stickee url en-AU,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Tutelary url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/OWAIS_NAEEM url en-US,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/averater url sv-SE,en,* php=zend
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Grind24 url fr-FR,en,* php=zend
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - php=zend
 www.text/..Mozilla/5.0 (compatible; GoogleBot/2.1 urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - php=zend
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - php=hhvm
 www.image/..Mozilla/5.0 (compatible; GoogleBot/2.1 urlGoogleBot.com/bot.html) - -
wikiwand
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=zend
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - php=hhvm
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - php=zend
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - php=zend
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en php=zend
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.24.4 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.18.1 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-216-g44cbbec (url) en php=zend
 scrapy.orgtext/..Scrapy/0.22.2 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-193-g7be3479 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.24.4 (url) en https=1;php=zend
 scrapy.orgtext/..Scrapy/0.24.2 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.25.0-214-gaa61f61 (url) en php=zend
 scrapy.orgtext/..Scrapy/0.16.5 (url) en php=zend
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - php=zend
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - php=zend
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - php=zend
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - php=hhvm
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.29 - php=zend
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.19 - -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - php=zend
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - php=zend
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - php=hhvm
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs php=zend
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) cs php=zend
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) - php=zend
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 php=zend
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - php=zend
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.98~bl; url) - php=zend
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot-SA/0.97; url) - php=zend
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - php=zend
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - php=zend
 tools.wmflabs.org/kmlexporttext/..url libwww-perl/6.03 - php=zend
 tools.wmflabs.org/geocommons/text/..url - php=zend
yacy
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.4-1-ARCH; java 1.7.0_67; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-32-generic; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-36-generic; java 1.7.0_65; Europe/fr) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-36-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/fr) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 8 6.2; java 1.8.0_11; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.16.2-gentoo; java 1.7.0_65; GMT/ja) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Linux 3.13.0-34-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-32-pve; java 1.7.0_03; Etc/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_32; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmlimage/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-1-amd64; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmlimage/..yacybot (/global; amd64 Linux 3.13.0-36-generic; java 1.7.0_65; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/nl) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-32-pve; java 1.7.0_65; America/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.41-042stab092.1; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab090.3; java 1.7.0_65; Etc/en) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) url en-us,en;q=0.5 php=zend
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 php=zend
cloudflare
 www.cloudflare.com/always-onlineimage/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
 www.cloudflare.com/always-onlinetext/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
unifiedlayer
 50-87-167-111.unifiedlayer.com/LaceyBuy/great2searchtext/..WordPress/4.0; url - php=zend
 173-254-124-63.unifiedlayer.com/LillianBesttext/..WordPress/4.0; url - php=zend
 173-254-50-252.unifiedlayer.com/TaliaTips/get2goinusatext/..WordPress/4.0; url - php=zend
 50-87-205-232.unifiedlayer.com/MadisonFindtext/..WordPress/4.0; url - php=zend
 50-87-201-249.unifiedlayer.com/LaurenToptext/..WordPress/4.0; url - php=zend
 50-87-201-32.unifiedlayer.com/AdrianaFasttext/..WordPress/4.0; url - php=zend
 50-87-201-249.unifiedlayer.com/HarperBesttext/..WordPress/4.0; url - php=zend
 50-87-0-170.unifiedlayer.com/RebeccaGuidetext/..WordPress/4.0; url - php=zend
 173-254-123-200.unifiedlayer.com/SarahGreattext/..WordPress/4.0; url - php=zend
 50-87-203-210.unifiedlayer.com/ShannonTopDealtext/..WordPress/4.0; url - php=zend
 173-254-99-209.unifiedlayer.com/MackenzieGurutext/..WordPress/4.0; url - php=zend
 50-87-201-102.unifiedlayer.com/NicoleSolutiontext/..WordPress/4.0; url - php=zend
 173-254-124-63.unifiedlayer.com/AshlynUnlocktext/..WordPress/4.0; url - php=zend
 50-87-201-249.unifiedlayer.com/AliviaOffer/bestinterioraccessoriestext/..WordPress/4.0; url - php=zend
 173-254-99-209.unifiedlayer.com/ScarlettGain/bestexterioraccessoriestext/..WordPress/4.0; url - php=zend
 173-254-127-17.unifiedlayer.com/SavannahSavetext/..WordPress/4.0; url - php=zend
 173-254-124-63.unifiedlayer.com/AshlynUnlock/bestdealreplacementpartstext/..WordPress/4.0; url - php=zend
 50-87-201-249.unifiedlayer.com/AliviaOffertext/..WordPress/4.0; url - php=zend
 50-87-205-232.unifiedlayer.com/GabrielaOnline/bestdealsperformanceparttext/..WordPress/4.0; url - php=zend
 50-87-203-21.unifiedlayer.com/KaitlynAcheivetext/..WordPress/4.0; url - php=zend
 173-254-103-171.unifiedlayer.com/amandausabestbuy2utext/..WordPress/4.0; url - php=zend
 173-254-99-209.unifiedlayer.com/ScarlettGaintext/..WordPress/4.0; url - php=zend
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - php=zend
plos
 alm.plos.orgapplication/jsonLagotto 3.6.3 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.5 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.6 - url - php=zend
 alm.plos.orgapplication/jsonLagotto 3.7 - url - php=zend
 sfo-stage-alm01.int.plos.orgapplication/jsonArticle-Level Metrics 3.4.8 - url - php=zend
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - php=zend
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - php=zend
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - php=zend
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 url) - php=zend
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - php=zend
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 php=zend
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - php=zend
wordpress
 goldendragon1971.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 goldendragon1971.wordpress.comtext/..WordPress/4.1-alpha-20141011; url - php=zend
 wikimediaukraine.wordpress.comtext/..WordPress/4.1-alpha-20141002; url - php=zend
 halokue.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 hoangkimlong.wordpress.comtext/..WordPress/4.1-alpha-20141011; url - php=zend
 hoangkimlong.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
 pronkpops.wordpress.comtext/..WordPress/4.0-RC3; url - php=zend
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - php=zend
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phpapplication/jsonTesting for url - php=zend
http:
 http://alm.example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - php=zend
 http://pkp-alm.lib.sfu.caapplication/jsonArticle-Level Metrics 3.6 - url - php=zend
flipboard
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 php=zend
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - php=zend
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - php=zend
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 php=zend
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-us php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-gb php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb php=zend
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address en-us
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.1.2 (url; mail address ru php=zend
etao
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - php=zend
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - php=zend
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; Please Name Your robot; url) - php=zend
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - php=zend
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - php=zend
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - php=zend
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - php=zend
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=zend
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch - php=zend
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - php=zend
xbmc
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.3;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.0.36 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Linux; Android; 3.10.33 armv7l; url) - -
example
 www.example.com/text/..wikidrain/1.0 (url) - php=zend
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - php=zend
 example.com/MyCoolTool/application/jsonMyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - php=zend
 example.com/MyCoolToolPage/text/..User-Agent: MyCoolTool (url) - php=zend
 example.com/MyCoolTool/text/..MyAwesomeImageDl/1.1 (url; mail address ) BasedOnSuperLib/1.4 - php=zend
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - php=zend
 www.feedly.com/fetcher.htmltext/..Feedly/1.0 (url; like FeedFetcher-Google) - php=zend
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en php=zend
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - php=zend
 www.majestic12.co.uk/bot.php?application/jsonMozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - php=zend
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - php=zend
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) -
zeerch
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - php=zend
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - php=zend
wesee
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PageBot (url) - php=zend
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - php=zend
 www.saciol.com/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - php=zend
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) - -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 php=zend
 www.diffbot.comimage/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
bluehost
 173-254-87-34.bluehost.com/PhoebeFasttext/..WordPress/4.0; url - php=zend
 173-254-123-200.bluehost.com/DaniellaDirecttext/..WordPress/4.0; url - php=zend
 173-254-99-91.bluehost.com/BuyandSave/patiolawngardentext/..WordPress/4.0; url - php=zend
 173-254-99-209.bluehost.com/CamrynUnlocktext/..WordPress/4.0; url - php=zend
 173-254-121-183.bluehost.com/rubyusabestbuy2utext/..WordPress/4.0; url - php=zend
kinshoo
 www.kinshoo.com/bot.htmltext/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 php=zend
 www.kinshoo.com/bot.htmlimage/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 php=zend
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - php=zend
 www.grapeshot.co.uk/crawler.phpimage/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
riddler
 riddler.io/about.htmltext/..Riddler (url) - php=zend
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=zend
 netarkivet.dk/webcrawlertext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=zend
 netarkivet.dk/webcrawlerimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.5.0-200506132127 url) - php=zend
drupal
 drupal.org/text/..Drupal (url) - php=zend
 drupal.org/application/jsonDrupal (url) - php=zend
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..Drupal (url) -
xip
crossref
 det.labs.crossref.orgapplication/jsonCrossRef DET 3.6 - url -
 det.labs.crossref.orgapplication/jsonCrossRef DET 3.6-beta - url -
 det.labs.crossref.orgapplication/jsonCrossRef DET 3.4.7 - url -
crazywebcrawler
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.0, url - php=zend
bne
 www.bne.es/es/LaBNE/ArchivoWeb/AvisoWebmasters/index.htmltext/..Mozilla/5.0 (compatible; heritrix/1.12.1 url) - php=zend
 www.bne.es/es/LaBNE/ArchivoWeb/AvisoWebmasters/index.htmlimage/..Mozilla/5.0 (compatible; heritrix/1.12.1 url) - -
uplabs0
 alm-1.uplabs0.comapplication/jsonArticle-Level Metrics 3.3.8 - url -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=zend
leiki
 www.leiki.comtext/..Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; url) - php=zend
searchingmotion
 searchingmotion.com/text/..SMcrawler (url) - php=zend
mysite
 www.mysite.com/application/vnd.php.serializedMyBot/1.0 (url) - php=zend
 www.mysite.com/text/..MyBot/1.0 (url) - php=zend
 www.mysite.com/application/jsonMyBot/1.0 (url) - php=zend
 www.mysite.com/botinfoapplication/vnd.php.serializedMozilla/5.0 (compatible; MyCoolBot/1.0; url) - php=zend
 www.mysite.comtext/..Mozilla/5.0 (compatible; heritrix/${pom.version} url) - php=zend
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - php=zend
datasift
 datasift.com/bot.htmltext/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - php=zend
 datasift.com/bot.htmlimage/..Mozilla/5.0 (TweetmemeBot/4.0; url) Gecko/20100101 Firefox/31.0 - -
test
 test.co.kr/text/..xxxxxxxx/Nutch-2.2.1 (url) - php=zend
github
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 php=zend
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - php=zend
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - php=zend
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - php=zend
 wiki.github.com/bixo/bixo/bixocrawlertext/..Mozilla/5.0 (compatible; pub-crawler; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
qbqm
 www.qbqm.fr/text/..QBQM/1.0 (url) fr php=zend
ac
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - php=zend
 www.clips.ua.ac.be/patterntext/..Pattern/2.6 url -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - php=zend
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - php=zend
 webarchiv.onb.ac.atimage/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
007ac9
 crawler.007ac9.net/text/..Mozilla/5.0 (compatible; 007ac9 Crawler; url) - php=zend
okian
 www.okian.ro/text/..MyBot/1.0 (url) - php=zend
 www.okian.ro/application/jsonMyBot/1.0 (url) - php=zend
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - php=zend
draigweb
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - php=zend
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - php=zend
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.3.0; url ) en-us,en;q=0.5 php=zend
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.2.0; url ) en-us,en;q=0.5 php=zend
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.3.0; url ) - php=zend
 OpenLinkProfiler.org/botapplication/jsonMozilla/5.0 (compatible; spbot/4.3.0; url ) - php=zend
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot; url) - php=zend
tineye
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - php=zend
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - php=zend
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - php=zend
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * php=zend
 archive-it.org/files/site-owners-special.htmltext/..Mozilla/5.0 (compatible; special_archiver; Archive-It; url) * php=zend
localhost
 localhost/clumpit/bot.phptext/..Mozilla/5.0 (compatible; clumboot; url) - php=zend
 localhostapplication/jsonLagotto 3.6.3 - url - php=zend
 localhost/wordpresstext/..WordPress/4.0; url - php=zend
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - php=zend
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
grouphigh
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - php=zend
 www.grouphigh.com/application/xmlMozilla/5.0 (compatible; GroupHigh/1.0; url) - php=zend
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - php=zend
SearchNearMe
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - php=zend
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - php=zend
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 php=zend
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - php=zend
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * php=zend
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - php=zend
recuweb
 recuweb.comapplication/jsonurl - php=zend
pinterest
 www.pinterest.com/text/..Pinterest/0.2 (url) - php=zend
 www.pinterest.com/image/..Pinterest/0.2 (url) - -
 pinterest.com/image/..Pinterest/0.1 url - -
wikidict
 www.wikidict.detext/..url - php=zend
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) - php=zend
netvibes
 www.netvibes.comtext/..Netvibes (url) - php=zend
 www.netvibes.comapplication/xmlNetvibes (url) - php=zend
 www.netvibes.comapplication/jsonNetvibes (url) en-us php=zend
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - php=zend
innercitylist
 classified.innercitylist.comtext/..WordPress/4.0; url - php=zend
 atlanta.innercitylist.comtext/..WordPress/4.0; url - php=zend
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - php=zend
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - php=zend
lagotto
 labs.lagotto.ioapplication/jsonLagotto 3.7-beta - url - php=zend
 labs.lagotto.ioapplication/jsonLagotto 3.7 - url - php=zend
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - php=zend
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - php=zend
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - php=zend
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - php=zend
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - php=zend
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - php=zend
nsidc
 nsidc.orgtext/..bcube.test.crawler/Nutch-1.6 (NSIDC BCUbe Web Service Search Engine; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
xovibot
 www.xovibot.net/text/..Mozilla/5.0 (compatible; XoviBot/2.0; url) - php=zend
crowdometer
 labs.crowdometer.orgapplication/jsonLagotto 3.6.3 - url - php=zend
 labs.crowdometer.orgapplication/jsonLagotto 3.6-beta - url - php=zend
 labs.crowdometer.orgapplication/jsonLagotto 3.6.2 - url - php=zend
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - php=zend
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - php=zend
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - php=zend
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
servicesonrequest
 www.servicesonrequest.comtext/..User-Agent: WikiBot (url) - php=zend
 www.servicesonrequest.com/application/jsonWikiBot(url) - php=zend
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - php=zend
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - php=zend
simplepie
 simplepie.orgtext/..SimplePie/1.3.1 (Feed Parser; url; Allow like Gecko) Build/20121030175403 - php=zend
 simplepie.orgapplication/xmlSimplePie/1.3.1 (Feed Parser; url; Allow like Gecko) Build/20121030175403 - php=zend
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - php=zend
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - php=zend
cdlib
 webarchives.cdlib.org/p/webmasterstext/..Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 url) - php=zend
 webarchives.cdlib.org/p/webmastersimage/..Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 url) - -
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - php=zend
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) it php=zend
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en php=zend
 bsurprised.com/text/..BSurprised WikiBox 0.1 (url) en php=zend
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) php=zend
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.13 (url) - php=zend
metamagazine
 metamagazine.comtext/..WordPress/3.9.2; url -
site
 www.site.com/application/jsonBot/2.0 (url) -
great-deal-spot
 great-deal-spot.comtext/..WordPress/4.0; url - php=zend
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - php=zend
 www.thinglink.com/help/ThinglinkImageBottext/..Elmer, the Thinglink ImageBot (url) - php=zend
 www.thinglink.com/help/ThinglinkImageBotimage/..Elmer, the Thinglink ImageBot (url) - -
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
perfectmarket
 www.perfectmarket.comtext/..Mozilla/5.0 (compatible; PerfectMarketKWTbot/1.0 url) - php=zend
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja php=zend
sciencecard
 dryad.sciencecard.orgapplication/jsonArticle-Level Metrics 3.6.3 - url - php=zend
inagist
 inagist.comtext/..InAGist URL Resolver (url) en-us,en;q=0.5 php=zend
site-shot
 www.site-shot.com/image/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Chromium/31.0.1650.63 Site-Shot/2.1 (url) Safari/534.34 en-US,* -
 www.site-shot.com/text/..Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Chromium/31.0.1650.63 Site-Shot/2.1 (url) Safari/534.34 en-US,* php=zend
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - php=zend
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - php=zend
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - php=zend
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - php=zend
edu
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - php=zend
 www.swust.edu.cntext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - php=zend
xxx
 www.xxx.comtext/..spider/Nutch-1.5.1 (spider; url) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - php=zend
 www.picsearch.com/bot.htmltext/..psbot-image (url) - php=zend
blogspot
 alberapps.blogspot.comtext/..TiempoBus/3.1.4 (url; mail address ) - php=zend
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 php=zend
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 php=zend
sprawk
 www.sprawk.com/text/..Sprawk/1.4 (url) - php=zend
thecardb
 thecardb.comtext/..theCarDB/1.1 (url; mail address ) - php=zend
brandprotect
 www.brandprotect.comtext/..LinkWalker/3.0 (url) - php=zend
thebesthomeproductsonline
 thebesthomeproductsonline.com/satellitetvtext/..WordPress/4.0; url - php=zend
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 php=zend
image2play
 image2play.com/image/..Mozilla/5.0 (compatible; Image2play/0.1; url) - -
verizon
 www.verizon.comapplication/jsonCrawler/1.0 (url; mail address ) Custom/1.0 - php=zend
hon
 www.hon.ch/honbot/text/..HonBot/0.01 ( mail address url) - php=zend
 www.hon.ch/honbot/text/..HonBot/0.01 ( mail address url) - -
wikilovesearth
 wikilovesearth.at/tools/text/..wle_at/1.0 (url; mail address ) - php=zend
QtWeb
import
 import.iotext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 php=zend
the-adam-green
 healthychocolatesonline.the-adam-green.nettext/..WordPress/3.9.2; url - php=zend
 giftbasketschocolate.the-adam-green.nettext/..WordPress/3.9.2; url - php=zend
livelap
 site.livelap.com/crawlertext/..LivelapBot/0.2 (url) en-US,en php=zend
192916.209999978total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - php=zend
 application/vnd.php.serialized
 text/..
 application/json
GoogleBot-Image/1.0 - -
 image/..
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
 application/pdf
MediaWikiCrawler-Google/2.0 ( mail address ) - php=zend
 text/..
php wikibot classes - php=zend
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN php=zend
 text/..
 application/xml
 application/json
LinkParser/2.0 - php=zend
 text/..
 application/pdf
py-radehh.py/rg10975 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
wikiwix-bot-3.0 - php=zend
 text/..
ClueBot/2.0 - php=zend
 application/vnd.php.serialized
 text/..
GoogleBot-Image/1.0 - php=zend
 text/..
 -
 image/..
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
welcome (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
pwb/rg3113 Pywikipediabot/2.0 - php=zend
 application/json
milog_bot/1.0 ( mail address ) - php=zend
 text/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
tigerbot - php=zend
 application/json
 text/..
pywikibot-compat-redirect.py/rg10993 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-radeh.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - php=zend
 application/json
 text/..
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 php=zend
 text/..
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
AniBot/0.9 php/curl - php=zend
 application/vnd.php.serialized
 text/..
maj_articles_recents (wikipedia:fr; User:ZéroBot) Pywikibot/2.0b2 (g4249) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
MediaWiki::Bot/3.2.6 - php=zend
 application/json
 text/..
Facebot/1.0 -
 text/..
 application/ogg
MediaWiki::Bot/5.005006 - php=zend
 application/json
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - php=zend
 application/json
 text/..
Twitterbot/1.0 - php=zend
 text/..
DigitalsmithsBot - php=zend
 text/..
maj_articles_recents/r3778 Pywikipediabot/2.0 - php=zend
 application/json
UAA-UAA.py/r10417 Pywikipediabot/1.0 - php=zend
 application/json
PythonWikipediaBot/1.0 - php=zend
 application/json
 text/..
 application/xml
mpandikasoratra/r-1 (unknown) Pywikibot/2.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11081 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - php=zend
 text/..
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - php=hhvm
 application/json
pywikibot-1newar25.py/rg11078 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Owlin Bot v. 3.0 - php=zend
 text/..
Screaming Frog SEO Spider/2.55 - php=zend
 text/..
 application/json
pywikibot-1newar25.py/rg11079 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11080 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Arachnida Web Crawler - php=zend
 text/..
DotNetWikiBot/3.11 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
pywikipedia-addgalleryurtest.py/r10860 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - php=zend
 application/json
MediaWiki::Bot/3.005002 - php=zend
 application/json
GoogleBot-Image/1.0 - php=zend
 text/..
 application/json
 image/..
pywikipedia-newinterwiki-fa.py/rg11077 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* php=zend
 text/..
 image/..
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; php=zend
 text/..
michiel_copylabel (wikidata:wikidata; User:RobotMichiel1972) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
spider - php=zend
 text/..
 application/json
 application/vnd.php.serialized
pywikipedia-git-desc_bot5.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
python-wikitools/1.2 (User:DickensBot) -
 application/json
WikidataBot framework - php=zend
 application/json
RBot/0.3 ( mail address ) -
 application/json
 image/..
 text/..
mussol-mussol.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
wymowa (commons:commons; User:Alkamid) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
SPIupdater-globalfunc.py/r10417 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1;) en-US,en;q=0.5 php=zend
 text/..
GoogleBot/GoogleBot 1.0 (GoogleBot; www.google.com; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
ifd-empty (wikidata:wikidata; User:PLbot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
 application/xml
ContextAd Bot 1.0 - php=zend
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - php=zend
 application/json
iw2data/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11078 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-git-del_new.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
Java Bot (https://github/vlsergey/Secretary; mail address ) BasedOnJWBF - php=zend
 text/..
 application/json
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - php=zend
 text/..
py-interwiki.py/rg11047 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-zzredirectyeh.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
 application/pdf
pywikipedia-zzfatranslator.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - php=zend
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - php=zend
 application/json
HTMLParser/2.0 - php=zend
 text/..
GoogleBot-Image/1.0 - https=1;php=zend
 text/..
Yeobot/1.1 Contact: mail address ) - php=zend
 text/..
Webbot - php=zend
 text/..
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - php=zend
 application/json
MyCuteBot/0.1 - php=zend
 text/..
 application/json
AnomieBOT/1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - php=hhvm
 application/json
cat_finder-core (wikipedia:fa; User:Rezabot) Pywikibot/2.0b2 (g4348) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11073 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlBot) - php=zend
 text/..
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
GermCrawler - php=zend
 application/json
 application/xml
 text/..
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
porzucone (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
MediaWiki::Bot/5.005007 - php=zend
 application/json
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - php=hhvm
 application/vnd.php.serialized
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - php=zend
 text/..
 application/json
Twitterbot/1.0 - -
 image/..
 text/..
 application/pdf
 application/ogg
 audio/webm
pywikibot-clean_sandbox.py/rg11078 Pywikipediabot/1.0 - php=zend
 application/json
Portale (wikipedia:it; User:TauerBot) Pywikibot/2.0b2 httplib2/0.7.4 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-git-desc_bot2.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
SineBot/1.5.19(User:SineBot) - php=zend
 application/vnd.php.serialized
 text/..
YisouSpider - php=zend
 text/..
 application/json
compat-checkimages.py/rs11055 Pywikipediabot/1.0 - php=zend
 application/json
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - php=zend
 application/json
Nutch Spider/Nutch-2.2.1 - php=zend
 text/..
BotReversor-BotReversor.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
editTerm/r2916 Pywikipediabot/2.0 - php=zend
 application/json
Spiderbot/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
pywikipedia-git-wdlabel.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
cat_finder-core (wikidata:wikidata; User:Rezabot) Pywikibot/2.0b2 (g4348) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
User-Agent Baiduspider - php=zend
 text/..
 image/..
pywikipedia-imageuncat.py/rg10972 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - php=hhvm
 application/json
 text/..
web-crawler en-US,en;q=0.8 php=zend
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
 application/xml
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us -
 image/..
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - php=zend
 text/..
 application/xml
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - php=zend
 text/..
 application/xml
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
pywikipedia-git-sectionBoten.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
Egerin Spider 1.0 - php=zend
 text/..
Mozilla/5.0 (compatible; OrangeBot/2.0; mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 php=zend
 text/..
WorldOfMusicBetaBot/1.0 - php=zend
 text/..
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - php=zend
 application/json
SLCrawl) - php=zend
 text/..
 application/opensearchdescription+xml
 application/rsd+xml
compat-basictemplatedisamblist.py/rg11081 Pywikipediabot/1.0 - php=zend
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.60; ) - php=zend
 text/..
 application/xml
pywikipedia-redirectsur.py/rg11078 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-newinterwiki-fa.py/rg11051 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
DotNetWikiBot/2.92 (Microsoft Windows NT 6.1.7600.0; ) - php=zend
 text/..
 application/xml
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
AnomieBOT/1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - php=hhvm
 application/json
pywikibot-compat-aaa_categorieAutori.py/rg10952 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
German Wikipedia Broken Weblinks Bot; contact: mail address - php=zend
 text/..
AnomieBOT/1.0 (TemplateSubster; see [[User:AnomieBOT]]) - php=hhvm
 application/json
PHPCrawl - php=zend
 text/..
 application/rsd+xml
DotNetWikiBot/2.96 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - php=zend
 text/..
cwikt-interwiki.py/rg11075 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pwb/rg3012 Pywikipediabot/2.0 - php=zend
 application/json
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
bot-07-fix-language-mappings (wikidata:wikidata; User:AkkakkBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
refreshfeeds (wikinews:ru; User:AKBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pwb/rg3009 Pywikipediabot/2.0 - php=zend
 application/json
compat-featured.py/r10441 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
run_pagefromfile.5/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
py-navbox.py/rg10975 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - php=zend
 application/json
run_pagefromfile.4/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
mgverbscript/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
GoogleBot (gocrawl v0.4) - php=zend
 text/..
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - php=zend
 text/..
Webwiki Search Engine Bot - www.webwiki.de - php=zend
 text/..
run_replace.7r.03/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
pywikibot-1newar25.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - php=zend
 application/vnd.php.serialized
 text/..
pywikipedia-commonscat.py/rg10961 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
wymowa (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=hhvm
 text/..
node-crawler/0.2.5 - php=zend
 text/..
inflectionof (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
 text/..
Facebot/1.0 - php=zend
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
cat_finder-core (wikipedia:fa; User:Rezabot) Pywikibot/2.0b2 (g4228) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 php=zend
 text/..
BOT/0.1 (BOT for JCE) - php=zend
 text/..
pywikipedia-git-desc_bot3.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikibot-1newar25.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
rapid-miner-crawler - php=zend
 text/..
ko_missing2/-1 (unknown) Pywikibot/2.0 (User:Chobot) - php=zend
 application/json
redirect (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
SentiBot www.sentibot.eu (compatible with GoogleBot) - php=zend
 text/..
php wikibot classes - php=hhvm
 application/vnd.php.serialized
 text/..
mrmetadata (wikipedia:en) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
EarwigBot/0.2.devgit-07bbf240 (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
pywikibot-compat-commonscat.py/rg11081 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
run_replace.7r.02/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
cwikt-interwiki.py/rg11078 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
compat-checkimages.py/rs11045 Pywikipediabot/1.0 - php=zend
 application/json
ifd-empty (wikidata:wikidata; User:PLbot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
SurakWare MediaWiki Bot/1.0 - php=zend
 text/..
pywikibot-compat-commonscat.py/rg11079 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
Goalkeeperbot(User:Beetstra)/1.0 - php=zend
 text/..
pywikibot-compat-commonscat.py/rg11070 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
GoogleBot/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
 application/opensearchdescription+xml
moogolbot/Moogol-1.0 (moogolbot) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
JackBot-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
Python27-pythonw.exe/r11079 Pywikipediabot/1.0 - php=zend
 application/json
PythonWikipediaBot/1.0 -
 application/json
 text/..
wymowa (commons:commons; User:Alkamid) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - php=zend
 text/..
 application/xml
pywikibot-compat-commonscat.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
py-radehh.py/rg10975 Pywikipediabot/1.0 - php=hhvm
 application/json
4615093 (wikipedia:en; User:RussBot) Pywikibot/2.0b2 (g4219) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
LinkParser/2.0 - php=hhvm
 text/..
pywikipedia-zzinter.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
HTMLParser/1.6 - php=zend
 text/..
pywiki-cfd.py/rg11032 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
michiel_copylabel (wikipedia:pl) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pwb/rg3536 Pywikibot/2.0 - php=zend
 application/json
RokaSpider/Nutch-1.8 (' mail address ') de-de,de,en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
nullref (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
CorenSearchBot/1.7 en libwww-perl/6.05 - php=zend
 text/..
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
Peachy MediaWiki Bot API Version 1.0 - php=zend
 application/vnd.php.serialized
t9/rg2816 Pywikipediabot/2.0 - php=zend
 application/json
MediaWikiCrawler-Google/2.0 ( mail address ) - php=hhvm
 text/..
michiel_copylabel (wikipedia:fr) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us php=zend
 text/..
AnomieBOT/1.0 (BAGBot; see [[User:AnomieBOT]]) - php=hhvm
 application/json
 text/..
CorenSearchBot/1.7 en libwww-perl/6.03 - php=zend
 text/..
XLinkBot/1.00 - php=zend
 text/..
compat-basicarticledisamblist.py/rg11081 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-radeh.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
Pastec bot -
 image/..
 text/..
pywikibot-compat-cfd.py/rg11060 Pywikipediabot/1.0 - php=zend
 application/json
NutchSpider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
addIdentifierMappingsProteinBoxBot/rg3599 Pywikibot/2.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11076 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
compat-redirect.py/rs11055 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
clean_sandbox (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
5003319 (wikipedia:en; User:RussBot) Pywikibot/2.0b2 (g4347) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
sport (wikipedia:fr; User:NaggoBot) Pywikibot/2.0b1 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
addDescTemplate/r2916 Pywikipediabot/2.0 - php=zend
 application/json
pywikipedia-css1.py/r10843 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Screaming Frog SEO Spider/2,55 - php=zend
 text/..
cat_finder-core (wikidata:wikidata; User:Rezabot) Pywikibot/2.0b2 (g4228) httplib2/0.8 Python/2.7.3.final.0 - php=zend
 application/json
ifd-hoobot (wikidata:wikidata; User:PLbot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
run_replace.7r.3/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
redirect/-1 (unknown) Pywikibot/2.0 (User:Rahl43bot) - php=zend
 application/json
pywikipedia-newar.py/rg11078 Pywikipediabot/1.0 - php=zend
 application/json
run_replace.7r.2/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
compat-redirect.py/rs11044 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
pywikipedia-newar.py/rg11079 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-sichteroptin.py/r10883 Pywikipediabot/1.0 - php=zend
 text/..
 application/json
SMcrawler-old-crawler - php=zend
 text/..
 application/opensearchdescription+xml
 application/rsd+xml
pywikipedia-zzgallery.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
pywikipedia-zzfatranslator.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
claimit (wikidata:wikidata; User:JAnDbot) Pywikibot/2.0b1 (s5217) httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
compat-checkimages.py/rs11056 Pywikipediabot/1.0 - php=zend
 application/json
My Bot - php=zend
 text/..
newitem (wikidata:wikidata; User:JAnDbot) Pywikibot/2.0b1 (s5217) httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
mrmetadata (wikipedia:de) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
Google Bot - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
pywikilab-llista_mils.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
(wikipedia:gl; User:Shyde) Pywikibot/2.0b1 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
BotKung/4.0 - php=zend
 text/..
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
Mozilla/5.0 (Apibot Browser) - php=zend
 application/vnd.php.serialized
compat-redirect.py/rs11060 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
 application/xml
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/json
UKBot [[:no:Bruker:UKBot]] - MwClient/0.7.0dev (https://github.com/mwclient/mwclient) - php=zend
 application/json
mesDemanats4/g3814 Pywikibot/2.0 (User:CastorBot) - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.3; ru-ru; Spider Build/JLS36C) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 ru-RU, en-US -
 image/..
DotNetWikiBot/3.10 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/json
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - php=zend
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; ) - php=zend
 text/..
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - php=zend
 text/..
michiel_copylabel (wikipedia:en) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
pywikipedia-commonscat.py/rg11078 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
ClueBot/2.0 - php=hhvm
 application/vnd.php.serialized
mrmetadata (wikipedia:it) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-zzpurging.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
bot-07-fix-language-mappings (wikidata:wikidata; User:AkkakkBot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; fr-fr; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 fr-FR, en-US -
 image/..
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - php=zend
 text/..
 application/xml
My Nutch Spider/Nutch-1.9 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
node.js - nodemw - mail address - Maintainer: Rainer Rillke - mail address - [[:commons:User:Rillke]] - php=zend
 application/json
python (wikipedia:cs; User:Frettiebot) Pywikibot/2.0b1 httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
wymowa (wiktionary:pl; User:AlkamidBot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
MoodleBot/1.0 - php=zend
 application/vnd.php.serialized
 application/xml
 text/..
pywikibot-1newar25.py/rg11070 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
gyffu_bot - php=zend
 text/..
removeDash/rg3599 Pywikibot/2.0 - php=zend
 application/json
4615093 (wikipedia:en; User:RussBot) Pywikibot/2.0b1 (g4211) httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-wikifyer.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Apibot Browser) - -
 image/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - php=zend
 text/..
 application/xml
 application/json
Botzilla-av_bot.py/rg11003 Pywikipediabot/1.0 - php=zend
 application/json
articles (wikipedia:fr; User:Peter17-Bot) Pywikibot/2.0b2 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-newar.py/rg11037 Pywikipediabot/1.0 - php=zend
 application/json
-c/-1 (unknown) Pywikibot/2.0 (User:RussBot) - php=zend
 application/json
coordinate_import/g3730 Pywikibot/2.0 (User:AinaliBot) - php=zend
 application/json
WikiBot/0.1 - php=zend
 text/..
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - php=hhvm
 application/json
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - php=zend
 application/xml
 text/..
refreshfeeds (wikinews:ru; User:AKBot) Pywikibot/2.0b1 httplib2/0.7.2 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia-zzaddnavebox.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
RSS Bot/1.9 (Mac OS X Version 10.10 (Build 14A389)) en;q=1 php=zend
 application/xml
 -
pywikibot-compat-commonscat.py/rg11073 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US php=zend
 text/..
 -
 image/..
 application/json
compat-redirect.py/rs11056 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
Lockalbot/0.0.1 - php=zend
 application/json
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 php=zend
 text/..
 application/json
Bot - php=zend
 text/..
welcome (wikinews:es; User:Ralgisbot) Pywikibot/2.0b1 (g4173) httplib2/0.9 Python/2.7.3.final.0 - php=hhvm
 application/json
AnomieBOT/1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - php=hhvm
 application/json
Inlibris.com XMLBot/1.0 - php=zend
 text/..
Mozilla/5.0 (compatible; SemanticBot/1.1) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
pywikipedia-zzinter.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
pagefromfile (wikisource:ru; User:HinoteBot) Pywikibot/2.0b1 (g4018) httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
RSS Bot/1.9 (Mac OS X Version 10.9.5 (Build 13F34)) en;q=1, ja;q=0.9, fr;q=0.8, de;q=0.7, es;q=0.6, it;q=0.5 php=zend
 application/xml
 -
cd-delinker.py/r11775 Pywikipediabot/1.0 - php=zend
 application/json
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
 text/..
pywikibot-compat-commonscat.py/rg11080 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
pywikipedia-zzfawikipatroler.py/rg11068 Pywikipediabot/1.0 - php=zend
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - php=zend
 text/..
CaBot - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US php=zend
 text/..
 -
 image/..
 application/json
pwb/rg3113 Pywikipediabot/2.0 - php=hhvm
 application/json
AnomieBOT/1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - php=hhvm
 application/json
KatBot bot.nytlabs.com> - php=zend
 text/..
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - php=hhvm
 application/json
pywikipedia-zzredirectyeh.py/rg11084 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
nullrefincat (wiktionary:en; User:MewBot) Pywikibot/2.0b2 httplib2/0.8 Python/2.7.6.final.0 - php=zend
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - php=hhvm
 application/json
compat-welcome.py/rs11039 Pywikipediabot/1.0 - php=zend
 application/json
pywikibot-compat-commonscat.py/rg11082 Pywikipediabot/1.0 - php=zend
 application/json
pywikipedia-addzumra.py/r10860 Pywikipediabot/1.0 - php=hhvm
 application/json
WibulBot - php=zend
 application/json
run/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
Elmyrbot/0.1 - php=zend
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
michiel_addlabel_pt (wikidata:wikidata; User:RobotMichiel1972) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
Wikibot/2.1.1 CFNetwork/672.1.15 Darwin/14.0.0 en-us -
 image/..
 text/..
python-wikitools/1.2 (User:Mr.Z-bot) - php=zend
 application/json
compat-checkimages.py/rs11057 Pywikipediabot/1.0 - php=zend
 application/json
Baiduspider - php=zend
 image/..
 text/..
michiel_copylabel (wikipedia:sv) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - php=zend
 text/..
 application/xml
MWFeedParser de-de php=zend
 application/xml
pywikipedia-bot_rss.py/rg14395 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
addMouseEntrezGeneInBatch/rg3599 Pywikibot/2.0 - php=zend
 application/json
pywikipedia-git-featured_new.py/rg90 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
 application/xml
pywikipedia-interwiki.py/rg26 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
milog_bot/1.0 ( mail address ) - php=hhvm
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
rookee-bot ru-ru,ru;q=0.5 php=zend
 text/..
 application/xml
Wikibot/2.1.1 CFNetwork/711.0.6 Darwin/14.0.0 en-us -
 image/..
 text/..
request-daemon/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
mrmetadata (wikipedia:uk) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikipedia3-welcome.py/r11747 Pywikipediabot/1.0 - php=zend
 application/json
MediaWiki::Bot/3.1.6 (User:SporkBot) - php=zend
 application/json
compat-redirect.py/rs11063 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
 application/xml
pywikibot-1newar25.py/rg11081 Pywikipediabot/1.0 - php=hhvm
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - php=zend
 text/..
pywikibot-compat-checkimages.py/rg10976 Pywikipediabot/1.0 - php=zend
 application/json
michiel_copylabel (wikipedia:de) Pywikibot/2.0b1 (4176) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
robot leapit.com en-US,* -
 image/..
recent/r3183 Pywikibot/2.0 - php=zend
 application/json
erfgoedbot-categorize_images.py/rg10979 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
iw_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
ko_missing/-1 (unknown) Pywikibot/2.0 (User:Chobot) - php=zend
 application/json
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
ntkbot/Nutch-1.8 en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
python (wikidata:wikidata; User:Frettiebot) Pywikibot/2.0b1 httplib2/0.9 Python/2.7.6.final.0 - php=zend
 application/json
MediaWiki::Bot/dev - php=zend
 application/json
WPBot 1.0 - php=zend
 text/..
GoogleBot-Video/1.0 - php=zend
 text/..
run_export/r-1 (unknown) Pywikipediabot/2.0 - php=zend
 application/json
Mozilla/5.0 (compatible; MojeekBot/0.6; https://www.mojeek.com/bot.html) - php=zend
 text/..
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 php=zend
 text/..
com.apple.Safari.WebFeedParser/600.1.4 CFNetwork/711.0.6 Darwin/14.0.0 en-us php=zend
 application/xml
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US php=zend
 -
 text/..
 image/..
 application/json
Python27-pythonw.exe/r10959 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
mrmetadata (wikipedia:ja) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
RavenCrawler - php=zend
 text/..
get_acteurs_Q (wikipedia:fr) Pywikibot/2.0b1 httplib2/0.8 Python/2.7.5.final.0 - php=zend
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US php=zend
 -
 text/..
 image/..
 application/json
vm-auto-erl-neu (wikipedia:de; User:Xqbot) Pywikibot/2.0b1 (s5183) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
pywikibot-compat-aaa_categorieTesti.py/rg10952 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
mrmetadata (wikipedia:id) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
Projet Sport-Projetsportliste2.py/r11052 Pywikipediabot/1.0 - php=zend
 application/json
 application/xml
delete_template (wikipedia:ru; User:WinterheartBot) Pywikibot/2.0b1 (g4196) httplib2/0.9 Python/2.7.7.final.0 - php=zend
 application/json
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - php=zend
 text/..
redirect (wikipedia:en; User:Xqbot) Pywikibot/2.0b2 (s5324) httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
AnomieBOT/1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - php=hhvm
 application/json
compat-pwb.py/rg11078 Pywikipediabot/1.0 - php=zend
 application/json
 text/..
mijnbots-Telstamboom.py/r-1 (unknown) Pywikipediabot/1.0 - php=zend
 application/xml
 application/json
a (wikipedia:mr; User:निनावी) Pywikibot/2.0b2 httplib2/0.9 Python/3.4.2.final.0 - php=zend
 application/json
mrmetadata (wikipedia:ru) Pywikibot/2.0b2 httplib2/0.9 Python/2.7.3.final.0 - php=zend
 application/json
maj_articles_recents (wikipedia:fr; User:ZéroBot) Pywikibot/2.0b2 (g4249) httplib2/0.7.2 Python/2.7.3.final.0 - php=hhvm
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1 - -
 image/..
 text/..
MoodleBot/1.0 -
 application/xml
 text/..
 application/vnd.php.serialized
32373.09total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Dec 8, 2014 14:24
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers