Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Jun 2014 - 30 Jun 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 144,538,630 page requests (mime type text/html only!) per day are considered crawler requests, out of 653,096,530 external requests, which is 22.1%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro007) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai07) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai04) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro006) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro005) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5 -
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai05) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5
 www.bing.com/bingbot.htmimage/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~doingfly) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai06) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 0.facebook.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; Facebookbot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.facebook.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 0.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 0.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/p/crawler4j/image/..crawler4j (url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mjatsoft-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - -
 code.google.com/p/ldspider/wiki/Robotstext/..ldspider (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~skawa-easyling) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~policevan2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp7) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - -
 code.google.com/p/gwtwiki/text/..JavaWikipediaAPI/3.0 url - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rxrwng) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ayicktigabelas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nashma4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp17) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vadim-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp15) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp12) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp18) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ungoodthinkful) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp19) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: batamsearch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp13) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: unblock4myspace) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~fernandolsa-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp11) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp14) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/p/ldspider/wiki/Robotsapplication/rdf+xmlldspider (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: minecraft6699) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp16) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nstahl2124) - -
 code.google.com/appengineapplication/jsonPython-urllib/2.7 AppEngine-Google; (url; appid: s~context-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: itechgiz-proxy-server) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yblk-ext-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nhsportal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zmeomzm) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: student-pupil-help) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: this-is-not-what-u-think) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~projection11111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: randohnson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~e-ddos) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy202) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gregatr0n) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mogoktharproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ikaryse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: red-arg) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ownsterserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~q-5278685c7a-x) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~no8lancelot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ydachao-3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beansacks) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~warawasetekure) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mistakeproxyarea) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usingomega) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitaniel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ahsanovic) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thegatetoweb) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lovelamp1988) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp7) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-data) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zmomyanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: predictionwrong) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yuricamara) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleBot/2.1; url) Safari/537.36 - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: madkaiser2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: navnode) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~alekproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kha1989led) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go6internet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kakuro69) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zoxy-1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kyaysarlay) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy10) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kurizogeorge) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azzaziprxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-mastered) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nuraiitj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vemmastar-c4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imabouncytigger) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp18) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: recrossed) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp19) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icecool1987) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: raja584sekhar) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp17) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kaveriselvaraj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ucsm111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp1) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davidgotmoney50) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~shanthuyeinmongnai) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp3) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp12) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: trabserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nation4india) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxies-list2) - -
 code.google.com/appengineapplication/jsonSyytacit AppEngine-Google; (url; appid: s~syytacit) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp11) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chronickg) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp6) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/rsd+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US https=1
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.2 - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai10) en-US -
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~ydachao-9) en-US -
 www.baidu.com/search/spider.htmlimage/..(compatible; Baiduspider/2.0; url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) fr-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) de-US, en-US -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) it-US, en-US -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) es-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) nl-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ja-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ru-US, en-US -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) pl-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) sv-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) tl-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) pt-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) vi-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) uk-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ca-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) no-US, en-US -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) id-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) fi-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) cs-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ko-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ar-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) tr-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) hu-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) he-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) fa-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) hr-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) sr-US, en-US -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ro-US, en-US -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) da-US, en-US -
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) bg-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ms-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) sk-US, en-US -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) lt-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) kk-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en,en-us;q=0.5 zero=470-01
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) et-US, en-US -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) sl-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) el-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible;GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) eu-US, en-US -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmlimage/..mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) eo-US, en-US -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) . en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url bn-BND,bn;q=0.5 zero=470-01
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) th-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) af-US, en-US -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozila/5.0(compitable;GoogleBot/2.1;=url en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url). en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ka-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) gl-US, en-US -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) az-US, en-US -
 www.google.com/bot.htmltext/..mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) be-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) uz-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) hy-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) lv-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) hi-US, en-US -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible;GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..mozilla/5.0(compatible;GoogleBot/2.1url) en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ky-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) mk-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en,en-us;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ta-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) la-US, en-US -
 www.google.com/bot.htmlimage/..Mozila/5.0 (compatible;GoogleBot/2.1;url) en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) oc-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) bn-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ur-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible;GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) sq-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en,bn-bnd;q=0.5 -
 www.google.com/text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url bot.html) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible;GoogleBot/2.1; url en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) is-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-gb,en;q=0.5 zero=470-01
 www.google.com/bot.htmlimage/..Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleBot/2.1; url) Safari/537.36 - -
 www.google.com/bot.html-Mozilla/5.0 (compatible;GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en,bn-bnd;q=0.5 -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
msn
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - https=1
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..(url) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) tr, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-US -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..(compatible; YandexImages/3.0; url) - -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de -
 cliqz.com/company/cliqzbotimage/..Cliqzbot/0.1 (url) de -
crossref
 alm.labs.crossref.orgapplication/jsonArticle Level Metrics 3.1.1 - url -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25(compatible;Sogou mobile spider/2.0;url) zh-cn -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
quickiwiki
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.10-11-desktop; java 1.8.0_05; Europe/es) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (allip-any; amd64 Linux 3.8.0-42-generic; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 R2 6.3; java 1.8.0_05; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; GMT08:00/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-29-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Windows Server 2012 R2 6.3; java 1.8.0_05; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-042stab088.4; java 1.6.0_31; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_31; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.14-1-amd64; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_31; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-29-generic; java 1.7.0_55; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.14-1-amd64; java 1.8.0_05; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.8.0-42-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_60; Europe/de) url en-us,en;q=0.5
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.14-1-amd64; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-30-generic; java 1.8.0_05; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.11.0-20-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12-1-amd64; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; i386 Linux 3.2.0-4-686-pae; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-65-generic; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.14.5-x86_64-linode42; java 1.7.0_55; Asia/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.14.1-gentoo; java 1.7.0_55; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0-27-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Windows 7 6.1; java 1.7.0_45; GMT-03:00/pt) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-042stab090.5; java 1.7.0_55; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_31; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 https=1
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_51; Asia/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.8.0-42-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13-1-amd64; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robottext/..(compatible; Exabot/3.0 (BiggerBetter); url)
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) - -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) * -
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.23.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.23.0 url -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Harry url en-US,* -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2image/..HTTP_Request2/0.5.2 (url) PHP/5.3.13p1 - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - -
naver
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 https=1
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
cloudflare
 www.cloudflare.com/always-onlineimage/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
example
 example.orgapplication/jsonArticle-Level Metrics 3.0 - url - -
 example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.orgapplication/jsonArticle-Level Metrics 3.2-beta - url - -
 example.comapplication/jsonArticle-Level Metrics - url - -
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-231-g2bf09b8 (url) en -
 scrapy.orgtext/..Scrapy/0.22.2 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-261-g7fd8a1a (url) en -
 scrapy.orgtext/..Scrapy/0.22.0 (url) en -
 scrapy.orgtext/..Scrapy/0.22.0 (url) de -
 scrapy.orgtext/..Scrapy/0.24.0 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
 scrapy.orgimage/..Scrapy/0.22.0 (url) en -
etao
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics 3.1.1 - url - -
pososhok
 pososhok.eu/application/jsonPososhok/1.1 (url; mail address ) Gecko/20100101 Firefox/25.0 - -
wordpress
 halokue.wordpress.comtext/..WordPress/4.0-alpha; url - -
 homeimprovementhomes.wordpress.comtext/..WordPress/4.0-alpha; url - -
 hoangkimlong.wordpress.comtext/..WordPress/4.0-alpha; url - -
 kterrl.wordpress.comtext/..WordPress/4.0-alpha; url - -
 pronkpops.wordpress.comtext/..WordPress/4.0-alpha; url - -
 zeitgeist77.wordpress.comtext/..WordPress/4.0-alpha; url - -
 drstvmovies.wordpress.comtext/..WordPress/4.0-alpha; url - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
flipboard
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ar -
yapq
 yapq.comapplication/jsonYapq mail address url - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; imagescoccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - -
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) - -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
 www.diffbot.comimage/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
 www.diffbot.comtext/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
sputnik
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) -
1eko
 roomshop.1eko.comtext/..WordPress/3.9.1; url - -
 volshop.1eko.comtext/..WordPress/3.9.1; url - -
 online1.roomshop.1eko.comtext/..WordPress/3.9.1; url - -
 online1.volshop.1eko.comtext/..WordPress/3.9.1; url - -
xbmc
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1; url) - -
 xbmc.orgtext/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.0 Git:20140505-f42f5b5 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.3;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.0 Git:20140505-f42f5b5 (Windows NT 6.1; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Linux; Android; 3.0.36 armv7l; url) - -
wikilovesearth
 wikilovesearth.at/tools/text/..wle_at/1.0 (url; mail address ) - -
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
http:
 http://alm.example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - -
github
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 -
 github.com/IvyBits/application/jsonWikiGraph/1.0 (url; mail address ) Python/idk - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
wikiwand
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140502-2251 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140502-2251 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
007ac9
 crawler.007ac9.net/text/..Mozilla/5.0 (compatible; 007ac9 Crawler; url) - -
 crawler.007ac9.net/text/..(compatible; 007ac9 Crawler; url) - -
wesee
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PageBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PictureBot (url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
site
 www.site.com/application/jsonBot/2.0 (url) -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/application/jsonMozilla/5.0 (compatible; BLEXBot/1.0; url) - -
toolserver
 toolserver.org/~dispenser/text/..DispensersTools (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) -
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) -
 toolserver.org/~dispenser/text/..CacheThumbs/1.4 (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug) - -
 toolserver.org/~dispenser/application/jsonDispensersTools (url) -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
 www.proximic.com/info/spider.phpimage/..Mozilla/5.0 (compatible; proximic; url) - -
riddler
 riddler.io/about.htmltext/..Riddler (url) - -
grouphigh
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - -
 www.grouphigh.com/bot.htmltext/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - -
 www.grouphigh.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GroupHigh/1.0; url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) en-us,en;q=0.5 -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
 www.saciol.com/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
dologo
 www.dologo.de/text/..Dologo Bot 1.0 (url)(61) - -
 www.dologo.de/text/..Dologo Bot 1.0 (url)(51) - -
 www.dologo.de/text/..Dologo Bot 1.0 (url)(52) - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
stolencamerafinder
 www.stolencamerafinder.com/text/..SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.stolencamerafinder.com/image/..SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
 www.okian.ro/application/jsonMyBot/1.0 (url) - -
wikidict
 www.wikidict.detext/..url - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) - -
 kc.nict.go.jp/project1/crawl.htmlimage/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
aaronsw
 www.aaronsw.com/2002/feedfinder/text/..feedfinder/1.371 Python-urllib/1.16 url - -
kinshoo
 www.kinshoo.com/bot.htmltext/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
bit
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) de
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) en
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
tineye
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) ja, en -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
draigweb
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url - -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
dataparksearch
 dataparksearch.org/bottext/..DataparkSearch/4.54-2014-01-09 (url) - -
 dataparksearch.org/bottext/..DataparkSearch/4.54-2014-01-09 (url) ru,en,fr,de -
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
Anonymouse
 Anonymouse.org/text/..url (Unix) - -
 Anonymouse.org/image/..url (Unix) - -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
 www.parsijoo.ir/image/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
blogspot
 beh1001.blogspot.com/p/the-bot.htmltext/..Mozilla/5.0 (compatible; Beh1001/beta; url) - -
 alberapps.blogspot.comtext/..TiempoBus/2.8 (url; mail address ) - -
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
mytest
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
zeerch
 lax1.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - -
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - -
 lax1.zeerch.com/bot.phpimage/..Mozilla/5.0 (compatible; LA1; url) - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
metamagazine
 metamagazine.comtext/..WordPress/3.9.1; url -
simplepie
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
idmarch
 www.idmarch.org/bot.htmltext/..Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; url) en-us,en-gb,en;q=0.7,*;q=0.3 -
qualitywatch
 www.qualitywatch.org/bottext/..Qualitywatch (url)(178) - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
temnos
 temnos.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) -
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.12 (url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
 crawler.sistrix.net/text/..(compatible; SISTRIX Crawler; url)
inagist
 inagist.comtext/..InAGist URL Resolver (url) en-us,en;q=0.5 -
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
twitter
 twitter.com/wiederfreitext/..wiederfreibot/1.0 (url) - -
smileforfood
 smileforfood.com/detext/..WordPress/3.9.1; url - -
 smileforfood.com/ittext/..WordPress/3.9.1; url - -
 smileforfood.com/uktext/..WordPress/3.9.1; url - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
enlle
 www.enlle.comtext/..Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; url; mail address ) es -
 www.enlle.comtext/..Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; url; mail address ) es, es;q=0.7 -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
queryseeker
 queryseeker.com/bot.htmltext/..QuerySeekerSpider ( url ) - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmltext/..psbot-image (url) - -
languagetool
 community.languagetool.orgapplication/xmlurl bot, contact: mail address - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
creativepulses
 creativepulses.nltext/..CreativePulses Crawler (url) - -
localhost
livedoor
 reader.livedoor.com/application/xmllivedoor FeedFetcher/0.01 (url; 1 subscriber) - -
 text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url www.google.com/ bot.html) en-us,en;q=0.5 zero=470-01
aol-soft
 www.aol-soft.com/text/..Mozilla/5.0 (compatible; AOLbot/4.0; url) en -
mysite
 www.mysite.com/application/jsonMyBot/1.0 (url) - -
 www.mysite.com/text/..MyBot/1.0 (url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en, hi -
srelease
 www.srelease.com/text/..Mozilla/5.0 (compatible; sReleaseBot/6.1; url) en -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot; url) - -
geolocation
 www.geolocation.wsapplication/jsonGeolocation.ws Wikimedia Commons import. url -
softlist
 www.softlist.us/text/..Mozilla/5.0 (compatible; SoftListBot/2.2; url) en -
QtWeb
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
 www.thinglink.com/help/ThinglinkImageBottext/..Elmer, the Thinglink ImageBot (url) - -
descargarprogramagratis
 www.descargarprogramagratis.com/text/..Mozilla/5.0 (compatible; PAD-bot/9.0; url) en -
194584.439999981total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/pdf
Peachy MediaWiki Bot API Version 2.0 (alpha 7) - -
 application/vnd.php.serialized
 text/..
 application/json
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - -
 application/vnd.php.serialized
 text/..
py-radehh.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
wikiwix-bot-3.0 - -
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
php wikibot classes - -
 application/vnd.php.serialized
 text/..
LinkParser/2.0 - -
 text/..
pwb/rg3113 Pywikipediabot/2.0 - -
 application/json
pwb/rg3428 Pywikibot/2.0 - -
 application/json
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
ClueBot/2.0 - -
 application/vnd.php.serialized
pywikipedia-rad1.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg3319 Pywikibot/2.0 - -
 application/json
 text/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - -
 application/json
pwb/rg3334 Pywikibot/2.0 - -
 application/json
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN -
 text/..
 image/..
 application/ogg
 application/pdf
 application/xml
 audio/x-flac
 audio/webm
 audio/midi
Facebot/1.0 -
 text/..
 image/..
 application/ogg
 video/webm
pwb/rg3295 Pywikibot/2.0 - -
 application/json
 text/..
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
tigerbot - -
 application/json
 text/..
 image/..
 application/ogg
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/json
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1 - -
 image/..
 text/..
 -
ProoXiBot fr -
 text/..
 application/xml
 application/ogg
PythonWikipediaBot/1.0 - -
 application/json
 text/..
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
maj_articles_recents/rg3186 Pywikibot/2.0 - -
 application/json
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
 application/ogg
 audio/webm
MediaWiki::Bot/5.005006 - -
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
pwb/rg2863 Pywikipediabot/2.0 - -
 application/json
 text/..
DigitalsmithsBot - -
 text/..
run_replace.7r/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pwb/rg3284 Pywikibot/2.0 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-1newar25.py/rg10995 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg10992 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg11013 Pywikipediabot/1.0 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
 application/json
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
MediaWiki::Bot/3.2.6 - -
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 application/ogg
pywikipedia-radeh.py/rg10961 Pywikipediabot/1.0 - -
 application/json
spider - -
 text/..
 application/vnd.php.serialized
 application/json
 image/..
 application/pdf
MediaWiki::Bot/3.005002 - -
 application/json
 text/..
pywikipedia-imageuncat.py/rg10972 Pywikipediabot/1.0 - -
 application/json
 application/xml
Webbot - -
 text/..
pywikipedia-zzfatranslator.py/rg10961 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25(compatible;SogouMobileSpider/2.0) zh-cn -
 text/..
 -
Owlin Bot v. 3.0 - -
 text/..
WikidataBot framework - -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - -
 application/json
 application/pdf
TrueKnowledgeBot bot mail address > - -
 application/xml
 application/vnd.php.serialized
 image/..
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - -
 text/..
 application/xml
maj_articles_recents/rg3395 Pywikibot/2.0 - -
 application/json
maj_articles_recents/rg3434 Pywikibot/2.0 - -
 application/json
pywikibot-compat-redirect.py/rg10993 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
wikiparser/1 CFNetwork/673.4 Darwin/13.2.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
compat-featured.py/rg10979 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-1newar25.py/rg11006 Pywikipediabot/1.0 - -
 application/json
 application/xml
Yeobot/1.1 Contact: mail address ) - -
 text/..
GoogleBot-Image/1.0 - https=1
 text/..
GermCrawler - -
 application/json
 text/..
 application/xml
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
ContextAd Bot 1.0 - -
 text/..
 image/..
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
CorenSearchBot/1.7 en libwww-perl/6.03 - -
 text/..
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-1newar25.py/rg10997 Pywikipediabot/1.0 - -
 application/json
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - -
 text/..
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-1newar25.py/rg10988 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg11010 Pywikipediabot/1.0 - -
 application/json
SecretaryBot (https://github/vlsergey/Secretary; mail address ) BasedOnJWBF - -
 text/..
pywikibot-1newar25.py/rg10990 Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-05-links-to-labels/rg3378 Pywikibot/2.0 - -
 application/json
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
BOT/0.1 (BOT for JCE) - -
 text/..
milog_bot/1.0 ( mail address ) - -
 text/..
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
pywiki-cfd.py/rg10926 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - -
 application/json
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
bot-05-links-to-labels/rg3381 Pywikibot/2.0 - -
 application/json
cwikt-interwiki.py/rg10992 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
compat-featured.py/r10441 Pywikipediabot/1.0 - -
 application/json
 application/xml
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
pywikibot-1newar25.py/rg11012 Pywikipediabot/1.0 - -
 application/json
Spinuf Spider - -
 text/..
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
maj_articles_recents/rg3382 Pywikibot/2.0 - -
 application/json
pywikipedia-zzredirectyeh.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzinter.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg3312 Pywikibot/2.0 - -
 application/json
Screaming Frog SEO Spider/2.40 - -
 text/..
 image/..
 application/json
pywikibot-1newar25.py/rg10986 Pywikipediabot/1.0 - -
 application/json
pywikipedia-newar.py/rg10992 Pywikipediabot/1.0 - -
 application/json
DanmicholoBot - -
 application/json
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-dave_potenziali_redirect_a_sezione.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Portale/rg3254 Pywikibot/2.0 - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pywikipedia-git-del_new.py/r90 Pywikipediabot/1.0 - -
 application/json
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-1newar25.py/rg10979 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzporb-main.py/rg10908 Pywikipediabot/1.0 - -
 application/json
KatBot bot.nytlabs.com> - -
 text/..
pwb/r2954 Pywikipediabot/2.0 - -
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
PythonWikipediaBot/1.0 -
 application/json
 text/..
compat-basictemplatedisamblist.py/rg10988 Pywikipediabot/1.0 - -
 application/json
 application/xml
YisouSpider - -
 text/..
 application/json
pwb/rs4287 Pywikibot/2.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
pywikibot-compat-aaa_categorieAutori.py/rg10952 Pywikipediabot/1.0 - -
 application/json
 application/xml
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 application/xml
run_replace.7r.3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-newinterwiki-fa.py/rg10993 Pywikipediabot/1.0 - -
 application/json
 application/xml
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
mgverbscript/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.1.14 Darwin/14.0.0 en-us -
 image/..
 text/..
logpatrol/rg3280 Pywikibot/2.0 - -
 application/json
ko_missing/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-newinterwiki-fa.py/r10334 Pywikipediabot/1.0 - -
 application/json
compat-basictemplatedisamblist.py/rg10976 Pywikipediabot/1.0 - -
 application/json
 application/xml
360spider-image - -
 image/..
 text/..
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-1newar25.py/rg10976 Pywikipediabot/1.0 - -
 application/json
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
pywikibot-1newar25.py/rg10987 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-checkimages.py/rs10958 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
bot-05-links-to-labels/rg3369 Pywikibot/2.0 - -
 application/json
cwikt-interwiki.py/rg10976 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rs4282 Pywikibot/2.0 - -
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
YBot/0.1 - -
 application/vnd.php.serialized
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - -
 application/json
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - -
 application/json
bot-05-links-to-labels/rg3360 Pywikibot/2.0 - -
 application/json
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 -
 application/json
GoogleBot-Image/1.0 zh-CN,zh;q=0.8,en;q=0.6 -
 image/..
 text/..
BotReversor-BotReversor.py/rg11003 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph112.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
compat-redirect.py/rs10967 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 image/..
 application/ogg
pywikipedia-git-wdph118.py/r90 Pywikipediabot/1.0 - -
 application/json
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-zztakhasosi.py/rg10908 Pywikipediabot/1.0 - -
 application/json
bot-05-links-to-labels/rg3416 Pywikibot/2.0 - -
 application/json
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
PHPCrawl - -
 text/..
porzucone/rg3440 Pywikibot/2.0 - -
 application/json
SentiBot www.sentibot.eu (compatible with GoogleBot) - -
 text/..
Erel Bot - -
 text/..
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
SurakWare MediaWiki Bot/1.0 - -
 text/..
bot-05-links-to-labels/rg3433 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - -
 text/..
 application/json
 application/xml
 application/rsd+xml
pywikibot-compat-commonscat.py/rg10979 Pywikipediabot/1.0 - -
 application/json
bot-05-links-to-labels/rg3387 Pywikibot/2.0 - -
 application/json
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
pwb/rg3292 Pywikibot/2.0 - -
 application/json
compat-featured.py/rg11012 Pywikipediabot/1.0 - -
 application/json
HTMLParser/1.6 - -
 text/..
German Wikipedia Broken Weblinks Bot; contact: mail address - -
 text/..
 application/ogg
pwb/rs4362 Pywikibot/2.0 - -
 application/json
 text/..
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-checkimages.py/rs10991 Pywikipediabot/1.0 - -
 application/json
compat-checkimages.py/rs10981 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - -
 text/..
pwb/rs4392 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
 -
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - -
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - -
 text/..
t9/rg2816 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
Mozilla/5.0 (compatible; SputnikBot/2.3) ru,en;q=0.1,*;q=0.01 -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
python-wikitools/1.2 (User:DickensBot) - -
 application/json
 text/..
pywikipedia-git-wdph116.py/r90 Pywikipediabot/1.0 - -
 application/json
XLinkBot/1.00 - -
 text/..
compat-basicarticledisamblist.py/rg10979 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-aaa_categorieTesti.py/rg10952 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-checkimages.py/rg10976 Pywikipediabot/1.0 - -
 application/json
py-navbox.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
refreshfeeds/rg3440 Pywikibot/2.0 - -
 application/json
compat-basicarticledisamblist.py/rg10990 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pywikibot-compat-pwb.py/rg10940 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
Livelapbot/0.1 en-US,en -
 text/..
 image/..
 application/pdf
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rs4306 Pywikibot/2.0 - -
 application/json
cd-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3334 Pywikibot/2.0 - -
 application/json
pwb/rs4394 Pywikibot/2.0 - -
 application/json
pywikipedia-zzhayati.py/rg10908 Pywikipediabot/1.0 - -
 application/json
COIBot/2.0 - -
 text/..
pywikipedia-git-coords.py/r90 Pywikipediabot/1.0 - -
 application/json
GjirafaSpider - -
 text/..
1409146/rg3297 Pywikibot/2.0 - -
 application/json
My Bot - -
 text/..
pywikipedia-featured.py/r(unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-newar.py/rg10972 Pywikipediabot/1.0 - -
 application/json
 application/xml
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
HTMLParser/2.0 - -
 text/..
 image/..
pywiki-lit.py/rg10993 Pywikipediabot/1.0 - -
 application/json
RSS Bot/1.8 (Mac OS X Version 10.9.3 (Build 13D65)) en;q=1, ja;q=0.9, fr;q=0.8, de;q=0.7, es;q=0.6, it;q=0.5 -
 application/xml
 -
porzucone/rg3334 Pywikibot/2.0 - -
 application/json
HRoestBot, de-wikipedia using pywikipedia framework -
 text/..
1761125/rg3400 Pywikibot/2.0 - -
 application/json
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
DotNetWikiBot/3.10 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
pywikipedia-commonscat.py/rg10972 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3254 Pywikibot/2.0 - -
 application/json
XoviBot/2.0 - -
 text/..
 image/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - -
 application/xml
 text/..
www.integromedb.org/Crawler - -
 text/..
 image/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
 application/json
GoogleBot-Image/1.0 bg -
 text/..
 image/..
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
 -
 application/json
pwb/rs4399 Pywikibot/2.0 - -
 application/json
pywikibot-compat-sichteroptin.py/r10883 Pywikipediabot/1.0 - -
 text/..
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
compat-basictemplatedisamblist.py/rg11006 Pywikipediabot/1.0 - -
 application/json
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10992 Pywikipediabot/1.0 - -
 application/json
replace_categ/r3183 Pywikibot/2.0 - -
 application/json
compat-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
pywikibot-1newar25.py/rg11016 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 -
 text/..
GoogleBot (gocrawl v0.4) - -
 text/..
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 - -
 application/json
 application/xml
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/pdf
pywikibot-compat-commonscat.py/rg10976 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg10985 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; SputnikImageBot/2.3) - -
 image/..
 text/..
compat-checkimages.py/rs10971 Pywikipediabot/1.0 - -
 application/json
MFU Spider(167)/Nutch-2.2.1 - -
 text/..
pywikibot-compat-commonscat.py/rg10990 Pywikipediabot/1.0 - -
 application/json
 application/xml
-c/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-featured.py/rg10990 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/rg11006 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/rg10988 Pywikipediabot/1.0 - -
 application/json
pywikipedia-commonscat.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-bot_rss.py/rg14302 Pywikipediabot/1.0 - -
 application/json
gsa-crawler-faa-soa-dev-one (Enterprise; T4-CPDF2HKLC8WX6; mail address , mail address ) - -
 text/..
pwb/r2772 Pywikipediabot/2.0 - -
 application/json
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 (https://github.com/mwclient/mwclient) - -
 application/json
compat-redirect.py/rs10954 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/rg3440 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 text/..
 -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
compat-basicarticledisamblist.py/rg11006 Pywikipediabot/1.0 - -
 application/json
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-aaa_listeAutori.py/rg10952 Pywikipediabot/1.0 - -
 application/json
COIBot/1.00 - -
 text/..
compat-featured.py/rg10997 Pywikipediabot/1.0 - -
 application/json
pywikipedia2-welcome.py/r10550 Pywikipediabot/1.0 - -
 application/json
pwb/rs4462 Pywikibot/2.0 - -
 application/json
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
pywikibot-compat-commonscat.py/rg11013 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
python-wikitools/1.2 (User:DickensBot) -
 application/json
niki-bot - -
 text/..
wikibot/r-1 (unknown) Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
pwb/rs4459 Pywikibot/2.0 - -
 application/json
OctraBot 3.3.3, OctraBot 3.3.3, Wiki.java 0.30(https://code.google.com/p/wiki-java/) - -
 text/..
pywikipedia-zzaddnavebox.py/rg10961 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3360 Pywikibot/2.0 - -
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
pwb/rg3366 Pywikibot/2.0 - -
 application/json
erfgoedbot-categorize_images.py/rg10979 Pywikipediabot/1.0 - -
 application/json
 application/xml
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
 application/json
pywikibot-1newar25.py/rg10993 Pywikipediabot/1.0 - -
 application/json
py-novbox.py/rg10975 Pywikipediabot/1.0 - -
 application/json
MoodleBot/1.0 - -
 text/..
 application/xml
 application/vnd.php.serialized
 image/..
 application/ogg
pwb/rg3381 Pywikibot/2.0 - -
 application/json
rijksmonumenten_import/rg3397 Pywikibot/2.0 - -
 application/json
com_end_space/rg3252 Pywikibot/2.0 - -
 application/json
porzucone/rg3416 Pywikibot/2.0 - -
 application/json
pwb/rs4375 Pywikibot/2.0 - -
 application/json
porzucone/rg3325 Pywikibot/2.0 - -
 application/json
porzucone/rg3338 Pywikibot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10981 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3443 Pywikibot/2.0 - -
 application/json
Screaming Frog SEO Spider/2.30 - -
 text/..
 image/..
bot-05-links-to-labels/rg3391 Pywikibot/2.0 - -
 application/json
porzucone/rg3264 Pywikibot/2.0 - -
 application/json
wikiparser/1 CFNetwork/673.4 Darwin/13.1.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 text/..
 image/..
Python27-pythonw.exe/r(unknown) Pywikipediabot/1.0 - -
 application/json
porzucone/rg3284 Pywikibot/2.0 - -
 application/json
pwb/rg3012 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
 -
 text/..
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3313 Pywikibot/2.0 - -
 application/json
porzucone/rg3366 Pywikibot/2.0 - -
 application/json
pywikipedia-rhmn_replace.py/r11781 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3456 Pywikibot/2.0 - -
 application/json
porzucone/rg3426 Pywikibot/2.0 - -
 application/json
porzucone/rg3280 Pywikibot/2.0 - -
 application/json
cdh-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3268 Pywikibot/2.0 - -
 application/json
compat-featured.py/rg11005 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3378 Pywikibot/2.0 - -
 application/json
porzucone/rg3297 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
archiv-erl-bot/rg2953 Pywikipediabot/2.0 - -
 application/json
articles/r2554 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3381 Pywikibot/2.0 - -
 application/json
porzucone/rg3343 Pywikibot/2.0 - -
 application/json
compat-checkimages.py/rs10969 Pywikipediabot/1.0 - -
 application/json
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/rg3391 Pywikibot/2.0 - -
 application/json
porzucone/rg3429 Pywikibot/2.0 - -
 application/json
porzucone/rg3387 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
porzucone/rg3395 Pywikibot/2.0 - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
pwb/rs4324 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (compatible; FlappyBot; ) - -
 text/..
 image/..
pywikibot-compat-commonscat.py/rg10988 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/rg10986 Pywikipediabot/1.0 - -
 application/json
pywikipedia-newar.py/rg10990 Pywikipediabot/1.0 - -
 application/json
PureBasic WikiBotFramework - d:User:InkoBot operated by d:User:Inkowik - -
 text/..
Mozilla/5.0 (compatible; SputnikImageBot/2.2) - -
 image/..
 text/..
Wegtam Crawler/Nutch-1.9-SNAPSHOT en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Inlibris.com XMLBot/1.0 - -
 text/..
pywiki-lit.py/rg10992 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3350 Pywikibot/2.0 - -
 application/json
request-daemon/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10995 Pywikipediabot/1.0 - -
 application/json
 application/xml
refreshfeeds/rg3381 Pywikibot/2.0 - -
 application/json
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3369 Pywikibot/2.0 - -
 application/json
porzucone/rg3433 Pywikibot/2.0 - -
 application/json
Crawler/1.2 - -
 text/..
compat-checkimages.py/rs10992 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1; en-US,en;q=0.5 -
 text/..
 image/..
wymowa/rg3381 Pywikibot/2.0 - -
 application/json
WikiBot/0.1 - -
 text/..
recent/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 text/..
SiocWikiBot/0.1 - -
 text/..
 application/vnd.php.serialized
pywikipedia-zzpurging.py/rg10961 Pywikipediabot/1.0 - -
 application/json
pwb/rs4454 Pywikibot/2.0 - -
 application/json
svn-compat-fixing_redirects.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/rg17 Pywikipediabot/1.0 - -
 application/json
 application/xml
refreshfeeds/rg3391 Pywikibot/2.0 - -
 application/json
pywikipedia-git-wdph122.py/r90 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3254 Pywikibot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10987 Pywikipediabot/1.0 - -
 application/json
compat-fr-tlfi.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
touch/r-1 (unknown) Pywikibot/2.0 - -
 application/json
python-wikitools/1.2 (User:LaraBot) -
 application/json
pwb/rs4479 Pywikibot/2.0 - -
 application/json
wymowa/rg3278 Pywikibot/2.0 - -
 application/json
Projet Sport-Projetsportliste2.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
 -
 application/json
wymowa/rg3350 Pywikibot/2.0 - -
 application/json
wymowa/rg3389 Pywikibot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg11006 Pywikipediabot/1.0 - -
 application/json
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 -
 application/json
Kolega2357-cosmetic_changes.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
-c/r2463 Pywikipediabot/2.0 - -
 application/json
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
commonscat/rg2781 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph126.py/r90 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 en-us -
 image/..
 text/..
MediaWiki::Bot/5.005007 - -
 application/json
wymowa/rg3431 Pywikibot/2.0 - -
 application/json
compat-de-duden.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
wymowa/rg3284 Pywikibot/2.0 - -
 application/json
wymowa/rg3424 Pywikibot/2.0 - -
 application/json
wymowa/rg3441 Pywikibot/2.0 - -
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
wymowa/rg3461 Pywikibot/2.0 - -
 application/json
Bot - -
 text/..
Mozilla/5.0 (compatible; MerchantCentricBot/1.0) en-US,* -
 text/..
wymowa/rg3456 Pywikibot/2.0 - -
 application/json
wymowa/rg3367 Pywikibot/2.0 - -
 application/json
ClueBot/2.0 (ClueBot NG Report Interface) - -
 text/..
wymowa/rg3312 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; fr-fr; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 fr-FR, en-US -
 image/..
 text/..
 -
OctraBot 3.3.3, OctraBot 3.3.3, OctraBot 3.3.3, Wiki.java 0.30(https://code.google.com/p/wiki-java/) - -
 application/json
 text/..
pwb/rg2777 Pywikipediabot/2.0 - -
 application/json
refreshfeeds/rg3387 Pywikibot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10997 Pywikipediabot/1.0 - -
 application/json
update-regular-users/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-1newar25.py/rg11011 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3319 Pywikibot/2.0 - -
 application/json
gsa-crawler (Enterprise; T4-KW9PXNCULXJSR; mail address ) - -
 text/..
pywikipedia-git-wdph111.py/r90 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3356 Pywikibot/2.0 - -
 application/json
wymowa/rg3373 Pywikibot/2.0 - -
 application/json
pywikipedia-checknewarticles_formaintenance-20130923.py/r11775 Pywikipediabot/1.0 - -
 application/json
refreshfeeds/rg3334 Pywikibot/2.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 -
 application/vnd.php.serialized
wymowa/rg3415 Pywikibot/2.0 - -
 application/json
pywikipedia-zzfawikipatroler.py/rg10961 Pywikipediabot/1.0 - -
 application/json
refreshfeeds/rg3431 Pywikibot/2.0 - -
 application/json
wymowa/rg3428 Pywikibot/2.0 - -
 application/json
my_bot/rg3152 Pywikipediabot/2.0 - -
 application/json
pywikipedia-flickrripper.py/r11489 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 GoogleBot/2.1 (Linux; Android/4.3; Galaxy Nexus Build/JZO54K) AppleWebKit/537.36 KHTML Chrome/28.0.1500.71 Mobile Safari/537.36 - -
 text/..
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko, Robot by TuiNu - -
 text/..
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
pywikipedia-git-new_iw14.py/r90 Pywikipediabot/1.0 - -
 application/json
pwb/rg3335 Pywikibot/2.0 - -
 application/json
TFAdefenderbot/rg3077 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3364 Pywikibot/2.0 - -
 application/json
34207.8899999999total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, Sep 19, 2014 16:51
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers