Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 May 2014 - 31 May 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 127,279,100 page requests (mime type text/html only!) per day are considered crawler requests, out of 641,848,320 external requests, which is 19.8%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; Facebookbot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 0.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 www.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; Facebookbot/2.1; url) en-US,en;q=0.5 -
 www.facebook.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 0.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5 -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~doingfly) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxycgi1-hrd) - -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/p/gwtwiki/text/..JavaWikipediaAPI/3.0 url - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 code.google.com/p/crawler4j/image/..crawler4j (url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/p/ldspider/wiki/Robotstext/..ldspider (url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2; rv:29.0) Gecko/20100101 Firefox/29.0 AppEngine-Google; (url; appid: s~ad0066a) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: predictionwrong) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/p/ldspider/wiki/Robotsapplication/rdf+xmlldspider (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mogoktharproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jerproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nuraiitj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azzaziprxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mjatsoft-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zmomyanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~policevan2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yblk-ext-proxy) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~projection11111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 www.google.com/bot.htmltext/..url en-US,en;q=0.5 zero=470-01
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ownsterserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn03) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: goeverysite) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nation4india) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beansacks) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imabouncytigger) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~e-ddos) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: this-is-not-what-u-think) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: raja584sekhar) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn03) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: clh-sw) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-myanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~prx600) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icecool1987) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khsprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thegatetoweb) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nhsportal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy202) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: randohnson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: suzetteklierocks) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: izzy-queenofthefae) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 KHTML Chrome/34.0.1847.131 Safari/537.36 AppEngine-Google; (url; appid: s~chinaastuceslp) fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: red-arg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~faproxyserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kyaysarlay) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: its-a-tr) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crowdsurfer100) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitalhunch-freenet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: student-pupil-help) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lovelamp1988) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kha1989led) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/27.0.1453.110 Safari/537.36 AppEngine-Google; (url; appid: s~livescorefeed-hrd) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sonydonin) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - https=1
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - https=1
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en,en-us;q=0.5 zero=470-01
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..- Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozila/5.0(compitable;GoogleBot/2.1;=url en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) fr-US, en-US -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) zh-cn -
 www.google.comtext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url /bot.html) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) de-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en,en-us;q=0.5 -
 www.google.com/bot.htmlimage/..Mozila/5.0(compitable;GoogleBot/2.1;=url en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url bot.html) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) it-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url bn-BND,bn;q=0.5 zero=470-01
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) es-US, en-US -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ja-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) pl-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) sv-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) ru-US, en-US -
 www.google.com/text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url bot.html) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible;GoogleBot/2.1;url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmltext/..Mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 zero=470-01
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) nl-US, en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible;GoogleBot/2.1; url en-US,en;q=0.5 zero=470-01
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en,en-us;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) pt-US, en-US -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-US, en-US -
 www.google.com/image/..Mozilla/5.0 (compatible; GoogleBot/2.1; url bot.html) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..mozilla/5.0 (compatible; GoogleBot/2.1 url) en-US,en;q=0.5 zero=470-01
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US -
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/rsd+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.2 - -
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmlimage/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai10) en-US -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - https=1
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
 search.msn.com/msnbot.htmapplication/jsonmsnbot-media/1.1 (url) - -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU
 yandex.com/botsimage/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25(compatible;Sogou mobile spider/2.0;url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13-1-amd64; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-openvz-042stab068.8; java 1.7.0_55; Europe/uk) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-openvz.28; java 1.6.0_25; Europe/uk) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-042stab088.4; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_31; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (allip/any; amd64 Linux 3.13-1-amd64; java 1.7.0_51; Europe/cs) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (allip/any; amd64 Linux 3.13-1-amd64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-61-generic; java 1.7.0_55; Europe/sv) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-61-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.13-1-686-pae; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.14.2-1-ARCH; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.11.10-11-desktop; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_55; Europe/nl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.14.1-gentoo; java 1.7.0_55; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0-27-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8.1 6.3; java 1.8.0_05; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.8.0-32-generic; java 1.7.0_25; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-29-pve; java 1.8.0_05; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-042stab084.12; java 1.6.0_31; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_17; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_51; Asia/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0-27-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-61-generic; java 1.6.0_31; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0; Europe/de) url en-us,en;q=0.5 -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robottext/..(compatible; Exabot/3.0 (BiggerBetter); url)
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.23.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.22.0 url - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.23.0 url -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
naver
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 https=1
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2image/..HTTP_Request2/0.5.2 (url) PHP/5.3.13p1 - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.19 - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 https=1
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.22.2 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-231-g2bf09b8 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-193-g5246986 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
 scrapy.orgtext/..Scrapy/0.18.1 (url) en -
 scrapy.orgtext/..Scrapy/0.16.3 (url) en -
 scrapy.orgtext/..Scrapy/0.22.0 (url) en -
bin-co
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
 nodejs.orgapplication/jsonNodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
example
 example.orgapplication/jsonArticle-Level Metrics 3.0 - url - -
 example.orgapplication/jsonArticle-Level Metrics 3.0-beta - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.comapplication/jsonArticle-Level Metrics - url - -
 example.orgapplication/jsonArticle-Level Metrics 2.14 - url - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
 tools.wmflabs.org/random-featured/application/vnd.php.serializedrandom-featured (Tool Labs; url) - -
cloudflare
 www.cloudflare.com/always-onlineimage/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
edu
 pjwstk.edu.pltext/..Wiki Controversial/0.1; url ; mail address -
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics 3.0 - url - -
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics 2.14 - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics 3.0-beta - url - -
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics 3.1.1 - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics 3.0 - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics 2.14 - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - -
wordpress
 halokue.wordpress.comtext/..WordPress/4.0-alpha; url - -
 homeimprovementhomes.wordpress.comtext/..WordPress/4.0-alpha; url - -
 goldendragonfamily.wordpress.comtext/..WordPress/4.0-alpha; url - -
 whatownsme.wordpress.comtext/..WordPress/4.0-alpha; url - -
 hoangkimlong.wordpress.comtext/..WordPress/4.0-alpha; url - -
 zeitgeist77.wordpress.comtext/..WordPress/4.0-alpha; url - -
 econstudentlog.wordpress.comtext/..WordPress/4.0-alpha; url - -
 drstvmovies.wordpress.comtext/..WordPress/4.0-alpha; url - -
 kingcrimsonprog.wordpress.comimage/..WordPress/4.0-alpha; url - -
flipboard
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
quickiwiki
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) ru -
 www.majestic12.co.uk/bot.php?application/jsonMozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
site
 www.site.com/application/jsonBot/2.0 (url) -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ru -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
yapq
 yapq.comapplication/jsonYapq mail address url - -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - -
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) - -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
 www.diffbot.comimage/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - -
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org bot; Wayback Machine Live Record; url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140502-2251 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 ( compatible; archive org bot; Wayback Live Record; url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url -
xbmc
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.0 Git:20140505-f42f5b5 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.0 Git:20140505-f42f5b5 (Windows NT 6.1; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130527-a36a92b (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Linux; Android; 3.0.36 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (iOS; 13.0.0, Version 6.1.4 (Build 10B809); url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.0 Git:20140505-f42f5b5 (Windows NT 6.3;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.2;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.0 Git:20140505-f42f5b5 (Linux; Android; 3.0.36 armv7l; url) - -
riddler
 riddler.io/about.htmltext/..Riddler (url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 www.tkl.iis.u-tokyo.ac.jp/~crawler/image/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
github
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/2.0.0p353 (url) en-us,en;q=0.5 -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
 www.proximic.com/info/spider.phpimage/..Mozilla/5.0 (compatible; proximic; url) - -
1eko
wikilovesearth
 wikilovesearth.at/tools/text/..wle_at/1.0 (url; mail address ) - -
mutat
 mutat.io/crawlertext/..Mozilla/5.0 (compatible; MutatioCrawler/0.1; url) - -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; ChinasoSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
toolserver
 toolserver.org/~dispenser/text/..CacheThumbs/1.4 (url) -
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) -
 toolserver.org/~dispenser/text/..DispensersTools (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/application/jsonDispensersTools (url) -
wesee
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PageBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PictureBot (url) - -
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.97~bl; url) - -
etao
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
http:
 http://alm.example.orgapplication/jsonArticle-Level Metrics 2.14 - url - -
 http://alm.example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
tineye
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
 www.okian.ro/application/jsonMyBot/1.0 (url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
 wiki.creativecommons.org/DiscoverEdtext/..DiscoverEd/Nutch-1.7 (OER search crawler; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
grouphigh
 www.grouphigh.com/bot.htmltext/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - -
 www.grouphigh.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GroupHigh/1.0; url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) ja, en -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) ja, en -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) - -
 kc.nict.go.jp/project1/crawl.htmlimage/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) en-us,en;q=0.5 -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
pososhok
 pososhok.eu/application/jsonPososhok/1.1 (url; mail address ) Gecko/20100101 Firefox/25.0 - -
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
aaronsw
 www.aaronsw.com/2002/feedfinder/text/..feedfinder/1.371 Python-urllib/1.16 url - -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
wikidict
 www.wikidict.detext/..url - -
bit
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) de
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) en
draigweb
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url - -
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url) -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
wikiwand
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
tu-darmstadt
 www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/text/..Mozilla/5.0 (compatible; kdslbot/3.2.0.5 url) - -
bigdatacorp
 ecommerce.bigdatacorp.com.br/faq.aspxtext/..Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; url) AppleWebKit/535.1 KHTML Chrome/13.0.782.107 Safari/535.1 pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4 -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
Anonymouse
 Anonymouse.org/text/..url (Unix) - -
 Anonymouse.org/image/..url (Unix) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
besaba
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
 text/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) urlGoogleBot.com Gecko/20100101 Firefox/23.0 en-us,en;q=0.5 -
 text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url www.google.com/ bot.html) en-US,en;q=0.5 zero=470-01
 text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url www.google.com/ bot.html) en-us,en;q=0.5 zero=470-01
 text/..Mozilla/5.0 (compatible; GoogleBot/2.1; url www.google.com/ bot.html) en-US,en;q=0.5 -
jeremysomer
 dev.jeremysomer.comtext/..WordPress/3.6; url - -
salefor
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
 watchmouse.com/text/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
blogspot
 beh1001.blogspot.com/p/the-bot.htmltext/..Mozilla/5.0 (compatible; Beh1001/beta; url) - -
 alberapps.blogspot.comtext/..TiempoBus/2.8 (url; mail address ) - -
petisud
 www.petisud.comtext/..WordPress/3.9.1; url - -
 www.petisud.comtext/..WordPress/3.9; url - -
webceo
 online.webceo.comtext/..Mozilla/5.0 (compatible; web-ceo-online-bot/1.0; url) - -
 online.webceo.comtext/..Mozilla/5.0 (compatible; web-ceo-online-bot/1.0; url) -
wandex
 wandex.nettext/..Mozilla/5.0 (compatible; World Wide Web Wanderer (Wandex Bot)/1.4; url) - -
nipponpaint
 nipponpaint.com.vntext/..WordPress/3.5.1; url - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
meximas
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
netnewswireapp
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
temnos
 temnos.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.12 (url) - -
simplepie
4
 110.164.210.4/bot.phptext/..Mozilla/5.0 (compatible; ZR2; url) - -
mytest
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
accelobot
 www.accelobot.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
enlle
 www.enlle.comtext/..Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; url; mail address ) es, es;q=0.7 -
 www.enlle.comtext/..Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; url; mail address ) - -
occultrevival
 occultrevival.nettext/..WordPress/3.9.1; url - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
inagist
 inagist.comtext/..InAGist URL Resolver (url) en-us,en;q=0.5 -
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en, hi -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
localhost
bne
 www.bne.es/es/LaBNE/ArchivoWeb/AvisoWebmasters/index.htmltext/..Mozilla/5.0 (compatible; heritrix/1.12.1 url) - -
filestream
 www.filestream.me/bot.htmltext/..Mozilla/5.0 (compatible; filestream/2.1; url) en-US,en;q=0.5 zero=470-01
 www.filestream.metext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 zero=470-01
creativepulses
 creativepulses.nltext/..CreativePulses Crawler (url) - -
QtWeb
profound
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) tr
scarecrow
 www.scarecrow.com/application/jsonScarecrow/1.0 (url; mail address ) - -
SearchNearMe
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
fu-berlin
 hpsg.fu-berlin.de/cowtext/..berlin-fu-cow (url) - -
 hpsg.fu-berlin.de/cowtext/..berlin-fu-cow/escow2014heuristicrawl (url) - -
 hpsg.fu-berlin.de/cowtext/..berlin-fu-cow/frcow2014heuristicrawl (url) - -
twitter
 twitter.com/wiederfreitext/..wiederfreibot/1.0 (url) - -
forecheck
 www.forecheck.comtext/..Forecheck (url) - -
queryseeker
 queryseeker.com/bot.htmltext/..QuerySeekerSpider ( url ) - -
brandpedia
 brandpedia.detext/..WordPress/3.9; url -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmltext/..psbot-image (url) - -
datoid
 datoid.cz/tag-information/application/jsonDatoid/0.9 (url; mail address ) BasedOnNette/2.0.9 - -
livedoor
 reader.livedoor.com/application/xmllivedoor FeedFetcher/0.01 (url; 1 subscriber) - -
com?type=714647&fr=spigot-yhp-ch
metamagazine
 metamagazine.comtext/..WordPress/3.9.1; url -
plagsniffer
 plagsniffer.de/application/jsonPlagsniffer (url) -
 plagsniffer.de/application/jsonPlagsniffer (url) - -
180574.459999975total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 7) - -
 application/vnd.php.serialized
 text/..
 application/json
pwb/rg3113 Pywikipediabot/2.0 - -
 application/json
 text/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 application/pdf
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
 application/pdf
 application/json
LinkParser/2.0 - -
 text/..
py-radehh.py/r10870 Pywikipediabot/1.0 - -
 application/json
 application/xml
wikiwix-bot-3.0 - -
 text/..
ClueBot/2.0 - -
 application/vnd.php.serialized
 text/..
pywikipedia-rad1.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
ProoXiBot fr -
 text/..
 application/ogg
 application/xml
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 application/xml
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - -
 application/json
 text/..
cis455crawler - -
 text/..
 image/..
 application/ogg
 application/pdf
 application/opensearchdescription+xml
 application/xml
pwb/rg2746 Pywikipediabot/2.0 - -
 application/json
tigerbot - -
 application/json
 text/..
 image/..
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
GoogleBot-Image/1.0 - -
 text/..
 image/..
 application/json
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
py-radehh.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
PythonWikipediaBot/1.0 - -
 application/json
 text/..
interwiki/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
MediaWiki::Bot/5.005006 - -
 application/json
Mozilla/5.0 (compatible; SputnikBot/2.3) ru,en;q=0.1,*;q=0.01 -
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN -
 text/..
 application/xml
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
 application/ogg
 video/webm
 audio/midi
maj_articles_recents/rg3077 Pywikipediabot/2.0 - -
 application/json
pywikibot-1newar25.py/rg10955 Pywikipediabot/1.0 - -
 application/json
 application/xml
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 application/ogg
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 text/..
DigitalsmithsBot - -
 text/..
TrueKnowledgeBot bot mail address > - -
 application/xml
 application/vnd.php.serialized
 image/..
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
 application/json
pywikibot-1newar25.py/rg10970 Pywikipediabot/1.0 - -
 application/json
 application/xml
CorenSearchBot/1.7 en libwww-perl/6.03 - -
 text/..
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
milcrawlbot/1.0 - -
 image/..
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
MediaWiki::Bot/3.2.6 - -
 application/json
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
maj_articles_recents/rg3012 Pywikipediabot/2.0 - -
 application/json
bot-05-links-to-labels/rg3093 Pywikipediabot/2.0 - -
 application/json
MediaWiki::Bot/3.005002 - -
 application/json
pwb/rg3046 Pywikipediabot/2.0 - -
 application/json
compat-featured.py/rg10957 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
maj_articles_recents/rg3186 Pywikibot/2.0 - -
 application/json
run_replace.7r/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
spider - -
 text/..
 application/json
 application/vnd.php.serialized
 application/pdf
My Nutch Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 application/ogg
 image/..
pywikipedia-newar.py/rg10961 Pywikipediabot/1.0 - -
 application/json
www.integromedb.org/Crawler - -
 text/..
 image/..
 application/ogg
pywikipedia-commonscat.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg3045 Pywikipediabot/2.0 - -
 application/json
chinesisch/r2916 Pywikipediabot/2.0 - -
 application/json
Webbot - -
 text/..
pwb/rg2823 Pywikipediabot/2.0 - -
 application/json
WikidataBot framework - -
 application/json
ContextAd Bot 1.0 - -
 text/..
 image/..
 audio/webm
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - -
 text/..
 application/xml
pywikipedia-radeh.py/rg10941 Pywikipediabot/1.0 - -
 application/json
 application/xml
BOT/0.1 (BOT for JCE) - -
 text/..
 application/json
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
pywikipedia-git-wdph107.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph111.py/r90 Pywikipediabot/1.0 - -
 application/json
wikiparser/1 CFNetwork/673.4 Darwin/13.1.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
pywikipedia-radeh.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg3067 Pywikipediabot/2.0 - -
 application/json
compat-newinterwiki-fa.py/r10334 Pywikipediabot/1.0 - -
 application/json
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzfatranslator.py/rg10961 Pywikipediabot/1.0 - -
 application/json
pwb/rg3108 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph108.py/r90 Pywikipediabot/1.0 - -
 application/json
GermCrawler - -
 application/json
 text/..
 application/xml
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
 audio/webm
 video/webm
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - -
 application/json
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Yeobot/1.1 Contact: mail address ) - -
 text/..
pywikibot-1newar25.py/rg10968 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
pwb/rg2916 Pywikipediabot/2.0 - -
 application/json
GjirafaSpider - -
 text/..
 image/..
pywikibot-1newar25.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
pwb/rs4070 Pywikipediabot/2.0 - -
 application/json
milog_bot/1.0 ( mail address ) - -
 text/..
GoogleBot-Image/1.0 - https=1
 text/..
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzfatranslator.py/rg10941 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10952 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
pywiki-cfd.py/rg10926 Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-05-links-to-labels/rg3046 Pywikipediabot/2.0 - -
 application/json
china/rg3170 Pywikipediabot/2.0 - -
 application/json
Portale/rg3123 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzredirectyeh.py/rg10908 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rs4074 Pywikipediabot/2.0 - -
 application/json
pywikipedia-radeh2.py/rg10941 Pywikipediabot/1.0 - -
 application/json
KatBot bot.nytlabs.com> - -
 text/..
pywikibot-1newar25.py/rg10956 Pywikipediabot/1.0 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
bot-07-fix-language-mappings/rg3081 Pywikipediabot/2.0 - -
 application/json
360spider-image - -
 image/..
 text/..
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
pwb/rs4254 Pywikibot/2.0 - -
 application/json
pwb/rs4169 Pywikipediabot/2.0 - -
 application/json
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-rename.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
SentiBot www.sentibot.eu (compatible with GoogleBot) - -
 text/..
gnd/r2916 Pywikipediabot/2.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
cwikt-interwiki.py/rg10965 Pywikipediabot/1.0 - -
 application/json
 application/xml
election/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
PythonWikipediaBot/1.0 -
 application/json
 text/..
pywikibot-compat-commonscat.py/rg10972 Pywikipediabot/1.0 - -
 application/json
 application/xml
web-crawler en-US,en;q=0.8 -
 text/..
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 -
 application/json
bot-07-fix-language-mappings/rg3055 Pywikipediabot/2.0 - -
 application/json
compat-featured.py/rg10960 Pywikipediabot/1.0 - -
 application/json
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 application/xml
Owlin Bot v. 3.0 - -
 text/..
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
compat-basictemplatedisamblist.py/rg10955 Pywikipediabot/1.0 - -
 application/json
 application/xml
WPBot 1.0 - -
 text/..
compat-basictemplatedisamblist.py/rg10972 Pywikipediabot/1.0 - -
 application/json
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - -
 application/json
compat-checkimages.py/rs10935 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-compat-commonscat.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg3082 Pywikipediabot/2.0 - -
 application/json
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
pywikibot-compat-aaa_categorieAutori.py/rg10952 Pywikipediabot/1.0 - -
 application/json
pwb/rg3075 Pywikipediabot/2.0 - -
 application/json
pwb/rs4148 Pywikipediabot/2.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.1.14 Darwin/14.0.0 en-us -
 image/..
 text/..
ko_missing/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
replace_empty_category/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pwb/rs4181 Pywikipediabot/2.0 - -
 application/json
add_category_deadend/rg2777 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 application/opensearchdescription+xml
pwb/r2772 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzporb-main.py/rg10908 Pywikipediabot/1.0 - -
 application/json
cis455crawler en -
 text/..
 image/..
 application/ogg
 application/pdf
pwb/rg3236 Pywikibot/2.0 - -
 application/json
compat-featured.py/rg10955 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 - -
 application/json
 application/xml
logpatrol/rg3050 Pywikipediabot/2.0 - -
 application/json
china/rg3141 Pywikipediabot/2.0 - -
 application/json
cwikt-interwiki.py/rg10947 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzinter.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 text/..
pwb/rs4101 Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
china/rg3152 Pywikipediabot/2.0 - -
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
pywikibot-1newar25.py/rg10965 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/r2954 Pywikipediabot/2.0 - -
 application/json
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pwb/rg2800 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
compat-checkimages.py/rs10951 Pywikipediabot/1.0 - -
 application/json
OctraBot 3.3.2, OctraBot 3.3.2, Wiki.java 0.30(https://code.google.com/p/wiki-java/) - -
 text/..
 application/json
pywikipedia-zzinter.py/rg10941 Pywikipediabot/1.0 - -
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - -
 text/..
pywikipedia-new_interwiki.py/r10843 Pywikipediabot/1.0 - -
 application/json
 application/xml
Erel Bot - -
 text/..
pwb/rs4108 Pywikipediabot/2.0 - -
 application/json
Innovazion Crawler/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
pywikibot-1newar25.py/rg10972 Pywikipediabot/1.0 - -
 application/json
XLinkBot/1.00 - -
 text/..
pywikibot-1newar25.py/rg10976 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/rg10972 Pywikipediabot/1.0 - -
 application/json
 application/xml
HTMLParser/2.0 - -
 text/..
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - -
 application/json
 text/..
SurakWare MediaWiki Bot/1.0 - -
 text/..
 application/xml
pwb/rs4198 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko, Robot by TuiNu - -
 text/..
pwb/rs4200 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
compat-checkimages.py/rs10930 Pywikipediabot/1.0 - -
 application/json
Polnet Parser v0.1, mail address -
 application/json
pywikipedia-git-maghalesaz.py/r90 Pywikipediabot/1.0 - -
 application/json
pwb/rg3081 Pywikipediabot/2.0 - -
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
categories_tree/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 image/..
 application/pdf
 application/ogg
Python27-pythonw.exe/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zztakhasosi.py/rg10908 Pywikipediabot/1.0 - -
 application/json
recent/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
German Wikipedia Broken Weblinks Bot; contact: mail address - -
 text/..
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
pywikipedia-git-wdph112.py/r90 Pywikipediabot/1.0 - -
 application/json
GoogleBot (gocrawl v0.4) - -
 text/..
pwb/rs4150 Pywikipediabot/2.0 - -
 application/json
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-del_new.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg10975 Pywikipediabot/1.0 - -
 application/json
pwb/r2816 Pywikipediabot/2.0 - -
 application/json
t9/rg2816 Pywikipediabot/2.0 - -
 application/json
HRoestBot, de-wikipedia using pywikipedia framework -
 text/..
 application/json
pwb/rs4193 Pywikipediabot/2.0 - -
 application/json
china/rg3181 Pywikibot/2.0 - -
 application/json
pywiki-newar25.py/rg10961 Pywikipediabot/1.0 - -
 application/json
pywikipedia-commonscat.py/rg10972 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-basicarticledisamblist.py/rg10975 Pywikipediabot/1.0 - -
 application/json
compat-basicarticledisamblist.py/rg10955 Pywikipediabot/1.0 - -
 application/json
compat-checkimages.py/rs10944 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/3.10 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
pywikipedia-newar.py/rg10970 Pywikipediabot/1.0 - -
 application/json
 application/xml
User-Agent Baiduspider - -
 text/..
 image/..
pywikibot-compat-commonscat.py/rg10965 Pywikipediabot/1.0 - -
 application/json
 application/xml
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pwb/rs4280 Pywikibot/2.0 - -
 application/json
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywiki-lit.py/rg10975 Pywikipediabot/1.0 - -
 application/json
pwb/rg3005 Pywikipediabot/2.0 - -
 application/json
bot-langmissing.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
aleph-avbot.py/r11368 Pywikipediabot/1.0 -
 application/json
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
porzucone/rg3046 Pywikipediabot/2.0 - -
 application/json
pwb/rs4136 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg10970 Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/rg3045 Pywikipediabot/2.0 - -
 application/json
YBot/0.1 - -
 application/vnd.php.serialized
1066076/rg3170 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-pwb.py/rg10940 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
718769/rg3067 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3170 Pywikipediabot/2.0 - -
 application/json
pwb/rs4059 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
Nutch Spider/Nutch-2.2.1 - -
 text/..
 image/..
pywikibot-compat-commonscat.py/rg10968 Pywikipediabot/1.0 - -
 application/json
My Bot - -
 text/..
tdrx-location-image-crawler: one-time crawl, about 100,000 articles; please contact mail address and temporarily block if this causes any inconvenience to the site. I am using a single thread. - -
 text/..
Livelapbot/0.1 en-US,en -
 text/..
 image/..
 application/pdf
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - -
 application/xml
 text/..
 application/atom+xml
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
pywikipedia-zzhayati.py/rg10908 Pywikipediabot/1.0 - -
 application/json
pwb/rs4265 Pywikibot/2.0 - -
 application/json
pwb/rs4129 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3141 Pywikipediabot/2.0 - -
 application/json
-c/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywiki-linktranslator.py/rg10959 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg10960 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3152 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/r10843 Pywikipediabot/1.0 - -
 application/json
ootu seed bot - -
 application/json
COIBot/2.0 - -
 text/..
pywikibot-compat-aaa_categorieTesti.py/rg10952 Pywikipediabot/1.0 - -
 application/json
 application/xml
GoogleBot-Image/1.0 bg -
 image/..
 text/..
cd-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 zh-CN,zh;q=0.8,en;q=0.6 -
 image/..
 text/..
py-novbox.py/r10870 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 -
 text/..
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 text/..
 -
Screaming Frog SEO Spider/2.40 - -
 text/..
 image/..
compat-checkimages.py/rs10954 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b - -
 text/..
 application/json
 image/..
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 (https://github.com/mwclient/mwclient) - -
 application/json
pwb/rs4067 Pywikipediabot/2.0 - -
 application/json
compat-redirect.py/rs10951 Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-07-fix-language-mappings/rg3046 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph113.py/r90 Pywikipediabot/1.0 - -
 application/json
pwb/rs4057 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-sichteroptin.py/r10883 Pywikipediabot/1.0 - -
 text/..
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 text/..
 image/..
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
wikibot-new-commonscat.py/rg10961 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1; en-US,en;q=0.5 -
 image/..
 text/..
china/rg3186 Pywikibot/2.0 - -
 application/json
pywiki-lit.py/rg10972 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
COIBot/1.00 - -
 text/..
wymowa/rg3046 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
Screaming Frog SEO Spider/2.30 - -
 text/..
 image/..
pwb/rs4096 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirectsur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-_move2wikidata_np_ru.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rs4103 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
compat-redirect.py/rs10936 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-newar.py/rg10972 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3045 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph116.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/rg10965 Pywikipediabot/1.0 - -
 application/json
pywikipedia2-welcome.py/r10550 Pywikipediabot/1.0 - -
 application/json
compat-redirect.py/rs10931 Pywikipediabot/1.0 - -
 application/json
 application/xml
YisouSpider - -
 text/..
 application/json
pwb/r3152 Pywikipediabot/2.0 - -
 application/json
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - -
 application/json
NutraspaceBot/Nutch-2.3-SNAPSHOT en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 application/ogg
pwb/rs4226 Pywikibot/2.0 - -
 application/json
/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - -
 text/..
 application/json
 application/xml
compat-featured.py/rg10952 Pywikipediabot/1.0 - -
 application/json
pywikipedia-commonscat.py/rg10908 Pywikipediabot/1.0 - -
 application/json
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
svn-compat-fixing_redirects.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 text/..
 image/..
 application/ogg
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 -
 application/json
PHPCrawl - -
 text/..
pywikipedia-commonscat.py/rg10970 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
addClaimFromTemplate/r2916 Pywikipediabot/2.0 - -
 application/json
pywiki-linktranslator.py/rg10956 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/rg13 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg2864 Pywikipediabot/2.0 - -
 application/json
ClueBot/2.0 (ClueBot NG Report Interface) - -
 text/..
porzucone/rg3042 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3095 Pywikipediabot/2.0 - -
 application/json
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-1newar25.py/rg10952 Pywikipediabot/1.0 - -
 application/json
HTMLParser/1.6 - -
 text/..
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
tdrx-location-image-crawler: one-time crawl, about 100,000 images; please contact mail address and temporarily block if this causes any inconvenience to the site. I am using a single thread. - -
 image/..
CaBot Script (running on nightshade.toolserver.org) - -
 application/vnd.php.serialized
pyreza-radeh2.py/rg10941 Pywikipediabot/1.0 - -
 application/json
pywikipedia-newar.py/rg10965 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph109.py/r90 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3039 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3172 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3077 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3122 Pywikipediabot/2.0 - -
 application/json
bot-07-fix-language-mappings/rg3226 Pywikibot/2.0 - -
 application/json
porzucone/rg3075 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
 text/..
 -
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
QrobotW - -
 text/..
porzucone/rg3067 Pywikipediabot/2.0 - -
 application/json
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - -
 text/..
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
EarwigBot/0.2.dev.git700b57de (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
MediaWiki::Bot/5.005007 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
pywikibot-compat-commonscat.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
cdh-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 text/..
 -
 application/json
pwb/rs4268 Pywikibot/2.0 - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
china/rg3226 Pywikibot/2.0 - -
 application/json
PureBasic WikiBotFramework - d:User:InkoBot operated by d:User:Inkowik - -
 text/..
pwb/rs4282 Pywikibot/2.0 - -
 application/json
porzucone/rg3119 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3174 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; waybackarchive.org/1.0; mail address ) de-de,de,en-us,en-gb,en;q=0.7,*;q=0.3
 text/..
logpatrol/rg3226 Pywikibot/2.0 - -
 application/json
wikibot-wpne-welcome.py/rg10970 Pywikipediabot/1.0 - -
 application/json
YourFilmsBot/0.1 - -
 application/json
OctraBot 3.3.1, OctraBot 3.3.1, Wiki.java 0.30(https://code.google.com/p/wiki-java/) - -
 text/..
 application/json
ITeco bot, v0.0, mail address - -
 text/..
porzucone/rg3165 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3186 Pywikibot/2.0 - -
 application/json
logpatrol/rg3011 Pywikipediabot/2.0 - -
 application/json
compat-featured.py/rg10956 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3108 Pywikipediabot/2.0 - -
 application/json
RBot/0.3 ( mail address ) -
 image/..
 application/json
 text/..
wikibot-new-commonscat.py/rg10970 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 -
 application/vnd.php.serialized
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywiki-newar25.py/rg10972 Pywikipediabot/1.0 - -
 application/json
file_has_rationale_yes/rg3102 Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.1.13 Darwin/14.0.0 en-us -
 image/..
 text/..
porzucone/rg3120 Pywikipediabot/2.0 - -
 application/json
porzucone/rg3068 Pywikipediabot/2.0 - -
 application/json
yacy - this is a proxy access through YaCy from a browser, not a robot (the yacy bot user agent is 'yacybot') - -
 text/..
 image/..
wikibot-wpne-welcome.py/rg10972 Pywikipediabot/1.0 - -
 application/json
bot-07-fix-language-mappings/rg3207 Pywikibot/2.0 - -
 application/json
pwb/rg3077 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1; en-US,en;q=0.5 zero=470-01
 text/..
pywikipedia-interwiki.py/rg16 Pywikipediabot/1.0 - -
 application/json
 application/xml
VBot 1.0.0.0 User:ValterVB - -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
wikibot-wpne-welcome.py/rg10961 Pywikipediabot/1.0 - -
 application/json
pywikipedia-checknewarticles_formaintenance-20130923.py/r11775 Pywikipediabot/1.0 - -
 application/json
NerdyBot - -
 text/..
pywikipedia-git-wdph106.py/r90 Pywikipediabot/1.0 - -
 application/json
refreshfeeds/rg3170 Pywikipediabot/2.0 - -
 application/json
python-wikitools/1.2 (User:LaraBot) -
 application/json
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
illustrate_wikidata/rg3039 Pywikipediabot/2.0 - -
 application/json
WikiBot/0.1 - -
 text/..
Inlibris.com XMLBot/1.0 - -
 text/..
wymowa/rg3039 Pywikipediabot/2.0 - -
 application/json
MapR Nutch Crawler/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
archiv-erl-bot/rg2951 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3122 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3174 Pywikipediabot/2.0 - -
 application/json
ci/rs4057 Pywikipediabot/2.0 - -
 application/json
wikibot-new-fixing_redirects.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg2777 Pywikipediabot/2.0 - -
 application/json
wikibot-wpne-commonscat.py/rg10961 Pywikipediabot/1.0 - -
 application/json
Projet IW-PIW2.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/rg3170 Pywikipediabot/2.0 - -
 application/json
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
pywikipedia-zzcity_article.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/rg3169 Pywikipediabot/2.0 - -
 application/json
xerador/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1 - -
 image/..
 text/..
 -
pwb/rg3073 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
china/rg3153 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3171 Pywikipediabot/2.0 - -
 application/json
update-regular-users/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
MySearch Spiderbot - -
 text/..
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 text/..
my_bot/r3152 Pywikipediabot/2.0 - -
 application/json
gsa-crawler (Enterprise; T4-KW9PXNCULXJSR; mail address ) - -
 text/..
refreshfeeds/rg3141 Pywikipediabot/2.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 en-us -
 image/..
 text/..
flickrAeroprints/rg3025 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3141 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3153 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3150 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3205 Pywikibot/2.0 - -
 application/json
wymowa/rg3120 Pywikipediabot/2.0 - -
 application/json
make_category_data_snapshot_earlier_finer/r2160 Pywikipediabot/2.0 - -
 application/json
refreshfeeds/rg3152 Pywikipediabot/2.0 - -
 application/json
wymowa/rg3081 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-new_iw14.py/r90 Pywikipediabot/1.0 - -
 application/json
35237.07total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Jun 16, 2014 14:36
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers