Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Nov 2013 - 30 Nov 2013 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 116,537,530 page requests (mime type text/html only!) per day are considered crawler requests, out of 654,856,400 external requests, which is 17.8%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..37.58.85.180 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5 -
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (Linux; Android/4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 KHTML Chrome/29.0.1025.133 Mobile Safari/535.19 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (Linux; Android/4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 KHTML Chrome/29.0.1025.133 Mobile Safari/535.19 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thepeterdeutsch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitaniel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: minecraft6699) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ucsm111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/p/gwtwiki/text/..JavaWikipediaAPI/3.0 url - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~dizykit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: puthiyathiravidan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beholdonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..Acre/dev/59:906 ubiquity.freebaseapps.com AppEngine-Google; (url; appid: s~freebaseapps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~warawasetekure) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: javahavens) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: easypox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thegatetoweb) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 KHTML Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (url; appid: s~feedly-social) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2; rv:24.0) Gecko/20100101 Firefox/24.0 AppEngine-Google; (url; appid: s~june0066y) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lovelamp1988) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kylesumrallproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vemmastar-c4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~certificate-check-403) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2; rv:24.0) Gecko/20100101 Firefox/24.0 AppEngine-Google; (url; appid: s~june0066d) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icecool1987) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~travelstaging) - -
 code.google.com/appenginetext/..Acre/dev/53:922 proxy.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jttm-server-prox) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icebre4ker) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gratistelcel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: anonchurro) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~tinyproxys) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: laaplicaciondelucas) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ageryder) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: labnol-server-proxy) - -
 www.google.com/coop/cse/creftext/..PageFetcher-Google-CoOp;((url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hoptheborder) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mhomeroxy) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nhsportal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lil05warrior) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~news-world-me) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ipreflector2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: etmproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl5) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..37.58.85.180 Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.0; url) en-us;q=0.7, en;q=0.3 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonGoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..37.58.85.180 Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/rsd+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh10) en-US -
 www.baidu.com/search/spider.htmltext/..188.129.143.73 Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8gg6) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~da8hao) en-US -
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh4) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh9) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~hhff2h) en-US -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~daff5hao) en-US -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..37.58.85.180 msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-Products/1.1 (url) - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botsapplication/vnd.php.serializedMozilla/5.0 (compatible; YandexBot/3.0; url) ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3 -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsapplication/pdfMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/text/..WikipediaApp_svc/1.5.13 (url; mail address ) com.orange.mmp/1.7.0 - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/application/jsonYeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..37.58.85.180 Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
http:
 http://alm.labs.crossref.orgapplication/jsonArticle Level Metrics - url - -
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/reader/faq/topic006/application/xmlMozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;url;1 subscribers;) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
wordpress
 radishmag.wordpress.comtext/..WordPress/3.8-alpha; url - -
 menschlein.wordpress.comtext/..WordPress/3.8-alpha; url - -
 y35pm.wordpress.comtext/..WordPress/3.8-alpha; url - -
 josefboberg.wordpress.comtext/..WordPress/3.8-alpha; url - -
 managra.wordpress.comtext/..WordPress/3.8-alpha; url - -
 kterrl.wordpress.comtext/..WordPress/3.8-alpha; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.8-alpha; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.7-beta2-25801; url - -
 tsjok45.wordpress.comtext/..WordPress/3.8-alpha; url - -
 whatownsme.wordpress.comtext/..WordPress/3.8-alpha; url - -
 dearkitty1.wordpress.comtext/..WordPress/3.8-alpha; url - -
 eversmarterworld.wordpress.comtext/..WordPress/3.8-alpha; url - -
 02varvara.wordpress.comtext/..WordPress/3.8-alpha; url - -
 tsjok45.wordpress.comtext/..WordPress/3.7-beta2-25801; url - -
 driwancybermuseum.wordpress.comtext/..WordPress/3.7-beta2-25801; url - -
 beyondthestarsastrology.wordpress.comtext/..WordPress/3.8-alpha; url - -
 halokue.wordpress.comtext/..WordPress/3.8-alpha; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.8-alpha; url - -
 dandelionsalad.wordpress.comtext/..WordPress/3.8-alpha; url - -
 ishfah7.wordpress.comtext/..WordPress/3.8-alpha; url - -
 hrexach.wordpress.comtext/..WordPress/3.8-alpha; url - -
 barbielistholland.wordpress.comtext/..WordPress/3.8-alpha; url - -
 melusinefee.wordpress.comtext/..WordPress/3.8-alpha; url - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_35; Europe/de) url - -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-56-generic; java 1.7.0_25; Europe/sv) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.9-xxxx-grs-ipv6-64; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.9-xxxx-grs-ipv6-64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.10.7-gentoo-r1; java 1.7.0_45; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; x86 Windows XP 5.1; java 1.7.0_25; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-54-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.7-gentoo-r1; java 1.7.0_45; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-54-generic; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.10-3-686-pae; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.2-031102-generic; java 1.7.0_25; America/es) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (superfi/global; amd64 Linux 2.6.32-042stab079.6; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.8; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.0-1-ARCH; java 1.7.0_45; Asia/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-1.16-desktop; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.0-1-ARCH; java 1.7.0_45; America/en) url en-us,en;q=0.5 -
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) - -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test2; url) * -
 fulltext.sblog.cz/screenshot/application/jsonMozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..(url) cs -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.27 - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) en-US,en;q=0.8,de;q=0.6,nb;q=0.4 -
 www.application/jsonGoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
toolserver
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) -
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/text/..CacheThumbs/1.4 (url) -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
wikipedia
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
bin-co
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
 www.majestic12.co.uk/bot.php?text/..(compatible; MJ12bot/v1.4.4; url) - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ru -
siteexplorer
 siteexplorer.info/text/..Mozilla/5.0 (compatible; SiteExplorer/1.0b; url) - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics - url - -
yahoo
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo-bot; url; mail address ) - -
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
 www.parsijoo.ir/image/..Mozilla/5.0 (compatible; parsijoo-bot; url; mail address ) - -
openwebspider
 www.openwebspider.org/text/..OpenWebSpider v0.1.4 (url) - -
 www.openwebspider.org/text/..JaKaBaBOT v0.1.4 (url) - -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 - -
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
wikidict
 www.wikidict.detext/..url - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
example
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 alm.example.orgapplication/jsonArticle-Level Metrics - url - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
 www.proximic.com/info/spider.phpimage/..Mozilla/5.0 (compatible; proximic; url) - -
scrapy
 scrapy.orgtext/..Scrapy/0.18.2 (url) en -
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.20.0 (url) en -
 scrapy.orgtext/..Scrapy/0.18.4 (url) en -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
yunyun
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/SiteInfo.php?r=aboutimage/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/SiteInfo.php?r=abouttext/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) - -
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
xbmc
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1; url) - -
microsystools
 www.microsystools.com/products/website-analyzer/text/..A1 Website Analyzer/5.0.1 (url) miggibot en-us,en,* -
 www.microsystools.com/products/website-analyzer/application/xmlA1 Website Analyzer/5.0.1 (url) miggibot en-us,en,* -
 www.microsystools.com/products/website-download/text/..A1 Website Download/3.0.3 (url) miggibot en-us,en,* -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
 tineye.com/crawler.htmlimage/..TinEye/1.1 (url) - -
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
 tineye.com/crawler.htmltext/..TinEye/1.1 (url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
kalooga
 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
owasp
 www.owasp.org/index.php/Category:OWASP_DirBuster_Projecttext/..DirBuster-0.12 (url) - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
seoprofiler
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0.1; url ) en-us,en;q=0.5 -
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0.2; url ) en-us,en;q=0.5 -
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0; url ) en-us,en;q=0.5 -
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0.2; url ) - -
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0.1; url ) - -
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0.3; url ) en-us,en;q=0.5 -
 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/4.0.3; url ) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
github
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.1 Ruby/2.0.0p247 (url) en-us,en;q=0.5 -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
 github.com/tenderlove/mechanize/image/..Mechanize/2.5.1 Ruby/1.8.7p352 (url) en-us,en;q=0.5 -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.3 Ruby/2.0.0p247 (url) en-us,en;q=0.5 -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
 www.grapeshot.co.uk/crawler.phpimage/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
gov
 linc.tas.gov.autext/..10.245.8.246 Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120320.221924 url) - -
 linc.tas.gov.autext/..10.245.8.245 Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120320.221924 url) - -
 linc.tas.gov.autext/..10.245.8.250 Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120320.221924 url) - -
 linc.tas.gov.autext/..10.245.8.252 Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120320.221924 url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xml-url en,hi -
ac
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
wesee
 www.wesee.com/en/support/bot/image/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/en/support/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
nvdev
 qa.nvdev.comtext/..Netvibes (url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
Anonymouse
 Anonymouse.org/text/..url (Unix) - -
 Anonymouse.org/image/..url (Unix) - -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
site
 www.site.com/application/jsonBot/2.0 (url) - -
netvibes
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) de-de -
 www.netvibes.comapplication/jsonNetvibes (url) fr-fr -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url) - -
holasoyramon
 www.holasoyramon.com/blogtext/..WordPress/3.5.1; url - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
automoveisnaweb
 automoveisnaweb.com.br/uk1text/..WordPress/3.7.1; url - -
 automoveisnaweb.com.br/uk2text/..WordPress/3.7.1; url - -
 automoveisnaweb.com.br/uk4text/..WordPress/3.6.1; url - -
 automoveisnaweb.com.br/uk3text/..WordPress/3.7.1; url - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (Macintosh; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
linkdex
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) - -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
yesup
 www.yesup.net/bot.htmltext/..Mozilla/5.0 (compatible; YesupBot/1.0; url) - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
metamagazine
 metamagazine.comtext/..WordPress/3.6; url - -
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
superfeedr
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.7 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.3 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.7 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.4 (url) - -
aramabeta
 www.aramabeta.comtext/..betasearch/Nutch-2.2.1 (Academic Beta Search; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
rebelmouse
 rebelmouse.comimage/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.comtext/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
com?type=714647&fr=spigot-yhp-ch
elefent
 elefent.eu/text/..elefent/Elefent 1.2 (A friendly web elefent.; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
imengine
 www.imengine.ittext/..Imengine/1.0 (url; mail address ) - -
netnewswireapp
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
 netnewswireapp.com/mac/-NetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
diffbot
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-US,en;q=0.8 -
simplepie
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) ja -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
wikimedia
 meta.wikimedia.org/wiki/User:Tietewtext/..Cheebot/0.5.7 (url) - -
 tools.wikimedia.de/~daniel/text/..WikiSense (url) - -
mytest
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131104-0244 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131104-0244 url) - -
gg
 gg.pl/text/..GG PeekBot 2.0 ( url http://info.gadu-gadu.pl/praca ) - -
 gg.pl/image/..GG PeekBot 2.0 ( url http://info.gadu-gadu.pl/praca ) - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) - -
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) de;q=0.7,*;q=0.3 -
nl
 was.nl.sgtext/..Mozilla/5.0 (compatible; NLB_archive_bot url National Library of Singapore, Web Archiving Project) - -
 was.nl.sgtext/..Mozilla/5.0 (compatible; NLB_archive_bot url) - -
 was.nl.sgtext/..Mozilla/5.0 (compatible; NLB_archive_bot url) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
qbqm
 www.qbqm.fr/text/..QBQM/1.0 (url) fr -
watchmouse
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
fueto
 fueto.comapplication/jsonFueto (url) - -
ibis
 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url) - -
 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url) - -
zeerch
 zx1.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; ZXBOT-ZX2; url) - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.10 (url) - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
sf
 liferea.sf.net/application/xmlLiferea/1.8.3 (Linux; fr_FR.UTF-8; url) - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
localhost
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
edu
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
profound
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - -
kr:6600
 www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsptext/..url - -
webdata
 webdata.co.intext/..WordPress/3.6.1; url - -
weserv
 images.weserv.nl/image/..Mozilla/5.0 (compatible; ImageFetcher/4.0; url) -
141269.099999983total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 5) - -
 application/vnd.php.serialized
 text/..
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
 application/ogg
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
LinkParser/2.0 - -
 text/..
redirect/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
tigerbot - -
 application/json
 text/..
 image/..
 -
 application/ogg
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
wikiwix-bot-3.0 - -
 text/..
 -
PythonWikipediaBot/1.0 - -
 application/json
 text/..
 application/xml
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
maj_articles_recents/r2217 Pywikipediabot/2.0 - -
 application/json
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
ClueBot/1.1 - -
 application/vnd.php.serialized
 text/..
WorldOfMusicBetaBot/1.0 - -
 text/..
pywikipedia-rad1.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; SputnikBot/2.2) ru,en;q=0.1,*;q=0.01 -
 text/..
 -
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 -
 text/..
maj_articles_recents/r2311 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
ClueBot/2.0 - -
 application/vnd.php.serialized
 -
 application/pdf
 text/..
maj_articles_recents/r2275 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 image/..
 application/xml
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
YisouSpider - -
 text/..
 -
 application/ogg
 application/xml
Semantix Bot 0.1 - -
 text/..
MediaWiki::Bot/3.2.6 - -
 application/json
 text/..
pywikipedia2-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 - -
 application/json
 text/..
MediaWiki::Bot/5.005006 - -
 application/json
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
BOT/0.1 (BOT for JCE) - -
 text/..
 -
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 text/..
 application/json
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 image/..
 application/json
 video/webm
 application/xml
 application/ogg
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 application/xml
 image/..
pywikipedia-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
DigitalsmithsBot - -
 text/..
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
compat-redirect.py/r10398 Pywikipediabot/1.0 - -
 application/json
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
 -
 audio/webm
 application/ogg
 video/webm
360spider-image - -
 image/..
 text/..
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10417 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-git-wdph661.py/r90 Pywikipediabot/1.0 - -
 application/json
German Wikipedia Dead Weblinks Checker Bot; contact: mail address - -
 text/..
 application/pdf
 image/..
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
pywikipedia-zzredirectyeh.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
strucr.com crawler 2.20.344 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
wikidata-coords_ghel.py/r10301 Pywikipediabot/1.0 - -
 application/json
 text/..
milog_bot/1.0 ( mail address ) - -
 text/..
moje-wymowa.py/r11775 Pywikipediabot/1.0 - -
 application/json
maj_articles_recents/r2299 Pywikipediabot/2.0 - -
 application/json
spider - -
 text/..
 application/json
 application/vnd.php.serialized
 image/..
compat-checkimages.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
 text/..
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 KHTML Chrome/24.0.1290.2 Safari/537.13 Lumibot/2.0 en-US,en;q=0.8 -
 image/..
 text/..
 application/json
MediaWiki::Bot/3.005002 - -
 application/json
 text/..
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
python-wikitools/1.2 (User:Mr.Z-bot) - -
 application/json
 text/..
new2-zzdi1.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
WikidataBot framework - -
 application/json
 text/..
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
GermCrawler - -
 application/json
 application/xml
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
hivaBot/hivaBot-0.2 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 application/ogg
 image/..
 application/xml
 video/webm
 audio/midi
pwb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
 application/pdf
add_category/r2219 Pywikipediabot/2.0 - -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
pywikipedia-git-wdph665.py/r90 Pywikipediabot/1.0 - -
 application/json
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-radeh.py/r10461 Pywikipediabot/1.0 - -
 application/json
 application/xml
botjagwar-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywiki-cfd.py/r10360 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 application/rsd+xml
 application/opensearchdescription+xml
AdMedia bot - -
 text/..
bot-newar.py/r10494 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
MyCuteBot/0.1 - -
 text/..
 application/json
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
compat-featured.py/r10334 Pywikipediabot/1.0 - -
 application/json
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
compat-pol.py/r10334 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 application/xml
pywikipedia-git-wdph663.py/r90 Pywikipediabot/1.0 - -
 application/json
pyreza-mianwi.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-git-wdph66.py/r90 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
 text/..
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
Webbot - -
 text/..
wikidata-sitelinks.py/r11618 Pywikipediabot/1.0 - -
 application/json
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.54; ) - -
 text/..
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 en-us -
 image/..
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
DanmicholoBot - -
 application/json
load_missed_pages/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
porzucone/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Peachy MediaWiki Bot API Version 2.0 (alpha 6) - -
 application/vnd.php.serialized
 text/..
spider/Nutch-1.4 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
compat-featured.py/r10398 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
compat-reportAirlinersUsage.py/r10494 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
gsa-crawler-ncr (Enterprise; T4-Q2WC86QM3C2W3; mail address , mail address ) - -
 text/..
 image/..
 -
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
wikiparser/1 CFNetwork/673.0.3 Darwin/13.0.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
internationalization/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Optimus5 Spider - -
 text/..
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 - -
 application/json
compat-basictemplatedisamblist.py/r10525 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzfatranslator.py/r10461 Pywikipediabot/1.0 - -
 application/json
glumik.bot/gbot-0.6 es-es,es-mx,es;q=0.7,*;q=0.3 -
 text/..
 image/..
www.monit24.pl-m24Bot/4.1- - -
 image/..
 text/..
fa-welcome.py/r11054 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikifyer.py/r10389 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-git-fromwdcat.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-interwiki.py/r10250 Pywikipediabot/1.0 - -
 application/xml
 application/json
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
voverb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
chrwikt/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pwb/r2219 Pywikipediabot/2.0 - -
 application/json
matilda/r2132 Pywikipediabot/2.0 - -
 application/json
harvest_books/r2160 Pywikipediabot/2.0 - -
 application/json
 text/..
pwb-sin_cat_import.py/r7 Pywikipediabot/1.0 - -
 application/json
bot1/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
COIBot/1.00 - -
 text/..
pywikipedia-wikidata_setproperty.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
pywikipedia-fr.wikt.RemiseEnForme.py/r68 Pywikipediabot/1.0 - -
 application/json
 application/xml
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - -
 application/json
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
mrajedrez-category_redirect.py/r11321 Pywikipediabot/1.0 - -
 text/..
 application/json
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
harvest_nltaxo/r2067 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-rdel2.py/r123 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zztakhasosi.py/r10269 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzinter.py/r10461 Pywikipediabot/1.0 - -
 application/json
 application/xml
HTMLParser/2.0 - -
 text/..
Mozilla/5.0 (compatible; MyBot/1.0;) - -
 application/json
pywikibot-compat-aaa_elenco.py/r10250 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
 image/..
SurakWare MediaWiki Bot/1.0 - -
 text/..
Test Webbot - -
 text/..
gsa-crawler (Enterprise; T4-DZLR68QD2A26J; mail address , mail address ) - -
 text/..
pywikipedia-touchf.py/r10389 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/r10444 Pywikipediabot/1.0 - -
 application/json
 application/xml
cwikt-interwiki.py/r10385 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en,* -
 image/..
 text/..
 application/json
pywikipedia-git-importwdcat.py/r90 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
 application/json
wAPI/1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
strucr.com crawler 2.21.346 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
compat-featured.py/r10525 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-wikidata_afb_getter.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
generate_plurals/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
DotNetWikiBot/2.103 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
pywikipedia-wikidata_setproperty_ID_frominfobox.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
ko_missing2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
ZeroBot - -
 text/..
pywikipedia-amb_mehrdadbot.py/r11775 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
Crawler/1.2 - -
 text/..
pywikirish-welcome.py/r11215 Pywikipediabot/1.0 - -
 application/json
ContextAd Bot 1.0 - -
 text/..
 image/..
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
aleph-avbot.py/r11368 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-box.py/r10461 Pywikipediabot/1.0 - -
 application/json
cwikt-interwiki.py/r10420 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/r10308 Pywikipediabot/1.0 - -
 application/json
 application/xml
botjagwar-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywikipedia-soccer.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
pywikipedia-wikifyer.py/r10461 Pywikipediabot/1.0 - -
 application/json
compat-afdbot.py/r10249 Pywikipediabot/1.0 - -
 application/json
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
 application/xml
 application/rsd+xml
compat-featured.py/r10441 Pywikipediabot/1.0 - -
 application/json
 application/xml
pd-crawler ( mail address ) - -
 text/..
MediaWiki::Perl/User:Wdwdbot - -
 application/json
pywikipedia-git-wdph662.py/r90 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 image/..
 -
 application/ogg
createHierarchy/r2067 Pywikipediabot/2.0 - -
 application/json
Spotmojo Bot 1.0 - -
 application/json
 text/..
 image/..
WPBot 1.0 - -
 text/..
 image/..
wikiparser/1 CFNetwork/596.5 Darwin/12.5.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
Python27-pythonw.exe/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-milhist.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
py-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - -
 application/xml
 text/..
Projet Sport-Projetsportliste2.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
pywikipedia-zzhayati.py/r10269 Pywikipediabot/1.0 - -
 application/json
cron/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
moje-porzucone.py/r11775 Pywikipediabot/1.0 - -
 application/json
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
compat-commonscat.py/r10385 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 text/..
 image/..
 application/json
SubjectAuthority-SubjectAuthority_0.1.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
wpbot-blockpageschecker.py/r-1 (unknown) Pywikipediabot/1.0 - -
 text/..
 application/json
3103658/r11657 Pywikipediabot/2.0 - -
 application/json
echooooooooooooo/r2090 Pywikipediabot/2.0 - -
 application/json
pywiki-newar25.py/r10380 Pywikipediabot/1.0 - -
 application/json
NikoBot1.0 - -
 text/..
 image/..
pywikipedia-com_mis_license_t.py/r10389 Pywikipediabot/1.0 - -
 application/json
XLinkBot/1.00 - -
 text/..
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
User-Agent Baiduspider - -
 text/..
 image/..
37.58.85.180 GoogleBot-Image/1.0 - -
 text/..
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
3180431/r11657 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph69.py/r90 Pywikipediabot/1.0 - -
 application/json
hardingbot - -
 text/..
commons-imagerecat.py/r11028 Pywikipediabot/1.0 - -
 application/json
botjagwar-getpron.py/r11215 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider; zh-CN -
 text/..
pywikipedia-pwb.py/r10377 Pywikipediabot/1.0 - -
 application/json
 text/..
 application/xml
 image/..
Mozilla/5.0 (X11; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3) de-de,en-us;q=0.7,en;q=0.3 -
 image/..
 text/..
 application/json
load_category/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
strucr.com crawler 2.20.343 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
ProjectDuke/Nutch-1.7 (the super crawl bot) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia-com_nl_pron.py/r10458 Pywikipediabot/1.0 - -
 application/json
 text/..
compat-featured.py/r10425 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10494 Pywikipediabot/1.0 - -
 application/json
 application/xml
count_genres/r2209 Pywikipediabot/2.0 - -
 application/json
GoogleBot - -
 text/..
 image/..
cd-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia-git-wdph664.py/r90 Pywikipediabot/1.0 - -
 application/json
Optimus5 Spider en-us,en;q=0.5 -
 text/..
pywikipedia-git-importwdcat2.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-imagerecat.py/r11775 Pywikipediabot/1.0 - -
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 text/..
compat-featured.py/r10458 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-rfubot.py/r10249 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-aaa_categorieAutori.py/r10250 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/r10540 Pywikipediabot/1.0 - -
 application/json
 application/xml
Wikibot/2.1.1 CFNetwork/672.0.2 Darwin/14.0.0 en-us -
 image/..
 text/..
erfgoedbot-categorize_images.py/r11633 Pywikipediabot/1.0 - -
 application/json
 application/xml
unknown, 2a00:4b00:13d:200::129c Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 text/..
 image/..
 application/json
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 text/..
 image/..
 -
 application/json
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10489 Pywikipediabot/1.0 - -
 application/json
QrobotW - -
 text/..
 image/..
unknown, 2a00:4b00:13d:200::129b Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 text/..
 image/..
 application/json
harvest_template/r2067 Pywikipediabot/2.0 - -
 application/json
compat-basicarticledisamblist.py/r10523 Pywikipediabot/1.0 - -
 application/json
pwb-klbot_wikiviajes_dmoz.py/r7 Pywikipediabot/1.0 - -
 application/json
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
Addbot Wikimedia Bot - -
 text/..
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
Inlibris.com XMLBot/1.0 - -
 text/..
pywikipedia-flickrripper.py/r11489 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/r10525 Pywikipediabot/1.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129d Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 text/..
 image/..
 application/json
pywikipedia-massblock.py/r10523 Pywikipediabot/1.0 - -
 application/json
calpoly wikibot - -
 application/json
 text/..
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
Mozilla/5.0 (compatible; Spiderlytics/1.0; mail address ) de-de,de,en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
unknown, 2a00:4b00:13d:200::129a Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
pywikipedia-com_mis_license_t.py/r10261 Pywikipediabot/1.0 - -
 application/json
pywikipedia-touchf.py/r10261 Pywikipediabot/1.0 - -
 application/json
 application/xml
archiv-erl-bot/r2203 Pywikipediabot/2.0 - -
 application/json
 text/..
unknown, 2a00:4b00:13d:200::129e Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 text/..
 image/..
 application/json
pywikipedia-checknewarticles_formaintenance-20130923.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_missinglabel_add2.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-com_nl_pron.py/r10389 Pywikipediabot/1.0 - -
 application/json
 text/..
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
pywikipedia-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
IssueCrawler - -
 text/..
DeletionBot-deletion.py/r10244 Pywikipediabot/1.0 - -
 application/json
 application/xml
wikiscore-MakeScoreTable.py/r11775 Pywikipediabot/1.0 - -
 application/json
count_genres/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129f Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 text/..
 image/..
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-git-roostmokhv2.0.py/r123 Pywikipediabot/1.0 - -
 application/json
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
ko_missing/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 text/..
YBot/0.1 - -
 application/vnd.php.serialized
refresh_merk/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.102 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/r2222 Pywikipediabot/2.0 - -
 application/json
 text/..
Empedia Bot - -
 text/..
dmashine-CleanupTematic.py/r10334 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 0.1beta - -
 application/vnd.php.serialized
AnomieBOT 1.0 (IFDCloser; see [[User:AnomieBOT]]) - -
 application/json
bot-newar.py/r10540 Pywikipediabot/1.0 - -
 application/json
Luxobot/2.0 (php; toolserver; Cropbot) mail address - -
 application/vnd.php.serialized
pywikipedia-git-wdph69.py/r123 Pywikipediabot/1.0 - -
 application/json
bitlybot - -
 text/..
 image/..
Projet IW-PIW2.py/r10533 Pywikipediabot/1.0 - -
 application/json
 application/xml
python-wikitools/1.2 (User:LaraBot) - -
 application/json
theWxitBot/0.1 - -
 application/json
AnomieBOT 1.0 (CHUUClerk; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Screaming Frog SEO Spider/2.21 - -
 text/..
 image/..
Wiki.java 0.28 r153 (OctraBot 3.1) - -
 text/..
 application/json
sample_historic/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
msnbot - -
 text/..
compat-basicarticledisamblist.py/r10493 Pywikipediabot/1.0 - -
 application/json
Erel Bot - -
 text/..
wikidata_label_serie_season/r2234 Pywikipediabot/2.0 - -
 application/json
py-newar25.py/r10461 Pywikipediabot/1.0 - -
 application/json
commons-nulledit.py/r11674 Pywikipediabot/1.0 - -
 application/json
 -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
"echocrawl 2.0" - -
 text/..
 application/ogg
 -
replace/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
User:Bilderbot (derivativeFX) by Luxo on the Toolserver / PHP - -
 application/vnd.php.serialized
 text/..
gsa-crawler (Enterprise; T3-GMDVWEKT9WSQZ; mail address ) - -
 text/..
 image/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
 application/rsd+xml
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 zh-cn -
 image/..
 text/..
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
Mozilla/5.0 (Windows NT 6.1; WOW64) Gecko/20100101 Firefox/20.0 Evidon ( mail address ) en-US,en;q=0.5 -
 image/..
 text/..
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
PercolateCrawler/3.1.30 ( mail address ) - -
 text/..
 image/..
pywikipedia-git-ship2.py/r90 Pywikipediabot/1.0 - -
 application/json
pwb/r2254 Pywikipediabot/2.0 - -
 application/json
TVersity Media Robot - -
 text/..
pywikipedia-replace.py/r11747 Pywikipediabot/1.0 - -
 text/..
 application/json
 application/xml
gsa-crawler (Enterprise; T4-KW9PXNCULXJSR; mail address ) - -
 text/..
 -
Py-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 image/..
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
 -
compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
logpatrol/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
commons-substituter.py/r11436 Pywikipediabot/1.0 - -
 application/json
WikiBot/0.1 - -
 text/..
Arkbot/0.1 alpha - -
 application/json
25387.75total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Thu, Dec 12, 2013 19:34
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers