Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Oct 2013 - 31 Oct 2013 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 129,047,520 page requests (mime type text/html only!) per day are considered crawler requests, out of 666,817,840 external requests, which is 19.4%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5 -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5 - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~doingfly) - -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..Acre/dev/59:906 ubiquity.freebaseapps.com AppEngine-Google; (url; appid: s~freebaseapps) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitaniel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/p/gwtwiki/text/..JavaWikipediaAPI/3.0 url - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..Acre/dev/53:918 factchecker.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: puthiyathiravidan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: easypox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: javahavens) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thepeterdeutsch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lovelamp1988) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beholdonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - -
 www.google.com/coop/cse/creftext/..PageFetcher-Google-CoOp;((url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vemmastar-c4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thegatetoweb) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ucsm111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~warawasetekure) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345proxy) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2; rv:24.0) Gecko/20100101 Firefox/24.0 AppEngine-Google; (url; appid: s~june0066y) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 KHTML Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (url; appid: s~feedly-social) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icebre4ker) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: etmproxy) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: semeout) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lil05warrior) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icecool1987) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: longbows-hideout) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: epvweb2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: labnol-server-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 code.google.com/p/gwtwiki/image/..JavaWikipediaAPI/3.0 url - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~alekproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: minecraft6699) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ahsanovic) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~news-world-me) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/27.0.1453.110 Safari/537.36 AppEngine-Google; (url; appid: s~livescorefeed-hrd) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nhsportal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: alex2610ps) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: this-is-not-what-u-think) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~no8lancelot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tom-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: maurandk-proxy) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gratistelcel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: laaplicaciondelucas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ipreflector2) - -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: s~libmuteki2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyproxy2884) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jptaravellahighschool) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kylesumrallproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tinyproxys) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxies-list2) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kauflog) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxworx) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: djsk-moon) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demowaiy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jttm-server-prox) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elliptical-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: funnyinternetwebsite) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mboharsik) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: default-testing) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: optusabuse) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.0; url) en-us;q=0.7, en;q=0.3 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/rsd+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh10) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8gg6) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: jpg8jpg) en-US -
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh4) en-US -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~da8hao) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..188.129.143.73 Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/text/..WikipediaApp_svc/1.5.13 (url; mail address ) com.orange.mmp/1.7.0 - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/application/xmlAhrefsBot.Feeds v0.1; url - -
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.1 (NHN Corp.; url) ja-JP,ja;q=0.8,en-US;q=0.6,en;q=0.4 -
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url) ja, *;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url) - -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
sblog
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) - -
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/application/jsonMozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/application/jsonMozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test2; url) * -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_40; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (domengo-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Etc/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-54-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86 Windows XP 5.1; java 1.7.0_25; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (superfi/global; amd64 Linux 2.6.32-042stab079.6; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_04; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-29-generic; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.4.4; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-042stab078.27; java 1.7.0_40; Etc/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.10-3-686-pae; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.9-xxxx-grs-ipv6-64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-1.16-desktop; java 1.7.0_40; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.9-xxxx-grs-ipv6-64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.5; java 1.6.0_51; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.40; java 1.6.0_43; GMT01:00/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_40; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_09; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.7-gentoo; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-31-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10-3-amd64; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-41-generic; java 1.6.0_27; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.10-3-686-pae; java 1.7.0_21; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.3-gentoo; java 1.7.0_40; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86_64 Mac OS X 10.8.4; java 1.6.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (domengo-any; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-358.6.2.el6.x86_64; java 1.6.0_24; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.10.5-axel-server; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-31-generic; java 1.6.0_27; America/es) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (domengo-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.8.5; java 1.6.0_65; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.4-201.fc19.x86_64; java 1.7.0_40; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_25; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (domengo/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.7-gentoo-r1; java 1.7.0_40; GMT01:00/pl) url en-us,en;q=0.5 -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou News Spider/4.0(url) zh-cn -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.27 - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
wordpress
 radishmag.wordpress.comtext/..WordPress/3.7-beta1-25639; url - -
 peoplesadvocacycouncil.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.7-beta2-25801; url - -
 driwancybermuseum.wordpress.comtext/..WordPress/3.7-beta1-25639; url - -
 zedie.wordpress.comtext/..WordPress/3.7-beta2-25801; url - -
 josefboberg.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.7-beta1-25639; url - -
 y35pm.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 02varvara.wordpress.comtext/..WordPress/3.7-beta1-25639; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 raymondpronk.wordpress.comtext/..WordPress/3.7-beta1-25639; url - -
 klausgauger.wordpress.comtext/..WordPress/3.7-beta1-25639; url - -
 radishmag.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 raymondpronk.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 driwancybermuseum.wordpress.comtext/..WordPress/3.7-beta2-25801; url - -
 klausgauger.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 at37.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.7-beta1-25639; url - -
 y35pm.wordpress.comtext/..WordPress/3.7-beta2-25801; url - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)/Nutch-1.4 en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.-GoogleBot/2.1 (urlGoogleBot.com/bot.html)/Nutch-1.4 en-us,en-gb,en;q=0.7,*;q=0.3 -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
toolserver
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) -
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~dispenser/image/..CacheThumbs/1.3 (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/text/..CacheThumbs/1.4 (url) -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
 nodejs.orgapplication/jsonNodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
i
 i.tv/application/jsonTV Captions (url) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
echonest
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmapplication/xmlMozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmimage/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address es-es -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ru -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics - url - -
siteexplorer
 siteexplorer.info/text/..Mozilla/5.0 (compatible; SiteExplorer/1.0b; url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo-bot; url; mail address ) - -
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
 www.parsijoo.ir/image/..Mozilla/5.0 (compatible; parsijoo-bot; url; mail address ) - -
 www.parsijoo.ir/image/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
crossref
 alm.labs.crossref.orgapplication/jsonArticle Level Metrics - url - -
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.97; url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
 www.proximic.com/info/spider.phpimage/..Mozilla/5.0 (compatible; proximic; url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 - -
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
 web.archive.comtext/..Web archive/Nutch-1.7 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; NLIL_IAHarvester url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
wikidict
 www.wikidict.detext/..url - -
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
dragonoff
 www.dragonoff.com/php/dev/WikifySong.phpapplication/vnd.php.serializedurl - -
tineye
 tineye.com/crawler.htmlimage/..TinEye/1.1 (url) - -
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
 tineye.com/crawler.htmltext/..TinEye/1.1 (url) - -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot url) - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
 www.muso.comapplication/xmlMozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
example
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 alm.example.orgapplication/jsonArticle-Level Metrics - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.com/MyCoolTool/text/..MyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
pchome
 www.pchome.com.tw/pchomebot.htmtext/..Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
openwebspider
 www.openwebspider.org/text/..OpenWebSpider v0.1.4 (url) - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
xbmc
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.2;WOW64;Win64;x64; url) - -
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 wummel.github.com/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.4; url) - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/2.0.0p247 (url) en-us,en;q=0.5 -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.1 Ruby/2.0.0p247 (url) en-us,en;q=0.5 -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) ja -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
yunyun
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/SiteInfo.php?r=aboutimage/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) - -
 www.yunyun.com/SiteInfo.php?r=abouttext/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
web
 www.web.nl/text/..web.nl spider/1.7; url - -
ac
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 oasis.go.krtext/..Mozilla/5.0 (compatible; mail address url) - -
 oasis.go.krimage/..Mozilla/5.0 (compatible; mail address url) - -
kalooga
 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
automoveisnaweb
 automoveisnaweb.com.br/uk1text/..WordPress/3.6.1; url - -
 automoveisnaweb.com.br/uk2text/..WordPress/3.7; url - -
 automoveisnaweb.com.br/uk4text/..WordPress/3.6.1; url - -
 automoveisnaweb.com.br/uk1text/..WordPress/3.7; url - -
 automoveisnaweb.com.br/uk3text/..WordPress/3.7; url - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
site
 www.site.com/application/jsonBot/2.0 (url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
wesee
 www.wesee.com/en/support/bot/image/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/en/support/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
zeerch
 zx1.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; ZXBOT-ZX2; url) - -
 zx1.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; ZEERCHBOT; url) - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (Macintosh; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
diffbot
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en,* -
linkdex
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url) - -
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
nvdev
 qa.nvdev.comtext/..Netvibes (url) - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
rebelmouse
 rebelmouse.comimage/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.comtext/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
metamagazine
 metamagazine.comtext/..WordPress/3.6; url - -
psu
 citeseerx.ist.psu.edutext/..citeseerxbot (compatible; heritrix/1.14.4 url) - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) - -
netvibes
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
 www.netvibes.comtext/..Netvibes (url) - -
holasoyramon
 www.holasoyramon.com/blogtext/..WordPress/3.5.1; url - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
superfeedr
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.com-Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
Anonymouse
 Anonymouse.org/text/..url (Unix) - -
 Anonymouse.org/image/..url (Unix) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) - -
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.7 (url) - -
 www.backgroundswitcher.com/image/..John's Background Switcher 4.4 (url) - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
localhost
 localhost/wordpresstext/..WordPress/3.6.1; url - -
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url) en-us,en;q=0.5 -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
watchmouse
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
scrapy
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
ibis
 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url) - -
 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url) - -
tt-rss
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
zapbot
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url) - -
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url) - -
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url) - -
edu
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
zite
 zite.comtext/..Mozilla/5.0 (compatible; woriobot mail address url) - -
kr:6600
 www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsptext/..url - -
156640.169999983total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 5) - -
 application/vnd.php.serialized
 text/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
 application/pdf
 application/ogg
 video/webm
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
wikiwix-bot-3.0 - -
 text/..
 -
tigerbot - -
 application/json
 text/..
 image/..
PythonWikipediaBot/1.0 - -
 application/json
 text/..
 application/xml
 -
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
redirect/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 text/..
LinkParser/2.0 - -
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
 application/pdf
pywikipedia-rad1.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
 application/xml
ClueBot/1.1 - -
 application/vnd.php.serialized
WikidataBot framework - -
 application/json
 text/..
YisouSpider - -
 text/..
 -
 application/ogg
 application/opensearchdescription+xml
 audio/webm
pywikipedia-wikidata_cleanlabel.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
spider - -
 application/vnd.php.serialized
 text/..
 application/json
 image/..
 -
pywikipedia2-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
ClueBot/2.0 - -
 application/vnd.php.serialized
 text/..
pywikipedia-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
MediaWiki::Bot/5.005006 - -
 application/json
 text/..
Semantix Bot 0.1 - -
 text/..
pywikipedia-wikidata_coordinaten_NL.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 - -
 application/json
 text/..
maj_articles_recents/r2160 Pywikipediabot/2.0 - -
 application/json
hivaBot/hivaBot-0.2 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
 application/ogg
 image/..
 application/xml
 unknown/unknown
 audio/midi
Mozilla/5.0 (compatible; SputnikBot/2.2) ru,en;q=0.1,*;q=0.01 -
 text/..
 -
MediaWiki::Bot/3.2.6 - -
 application/json
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 image/..
 application/xml
 video/webm
DigitalsmithsBot - -
 text/..
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
maj_articles_recents/r2185 Pywikipediabot/2.0 - -
 application/json
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
 text/..
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 text/..
BOT/0.1 (BOT for JCE) - -
 text/..
 -
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
 application/ogg
Addbot Wikimedia Bot - -
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 text/..
 application/json
 -
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Keybot Translation-Search-Machine - -
 text/..
Mozilla/5.0 (compatible; SearchBot) ru,en;q=0.1,*;q=0.01 -
 text/..
 -
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
360spider-image - -
 image/..
 text/..
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
milog_bot/1.0 ( mail address ) - -
 text/..
moje-wymowa.py/r11483 Pywikipediabot/1.0 - -
 application/json
AarghBot Linux - -
 text/..
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
Web Crawler - -
 text/..
maj_articles_recents/r2108 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 application/rsd+xml
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 KHTML Chrome/24.0.1290.2 Safari/537.13 Lumibot/2.0 en-US,en;q=0.8 -
 image/..
 text/..
 application/json
 application/ogg
MediaWiki::Bot/3.005002 - -
 application/json
 text/..
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
pywikipedia-touchf.py/r10261 Pywikipediabot/1.0 - -
 application/json
 application/xml
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
GermCrawler - -
 application/json
 text/..
 application/xml
DanmicholoBot - -
 application/json
Test Webbot - -
 text/..
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
maj_articles_recents/r2213 Pywikipediabot/2.0 - -
 application/json
compat-checkimages.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
matilda/r2132 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
 application/json
 image/..
 application/ogg
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
pywikipedia-wikidata_descr.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 image/..
 application/ogg
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
compat-newinterwiki-ar.py/r10334 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
compat-featured.py/r10308 Pywikipediabot/1.0 - -
 application/json
 application/xml
WorldOfMusicBetaBot/1.0 - -
 text/..
moje-porzucone.py/r11483 Pywikipediabot/1.0 - -
 application/json
wikidata-sitelinks.py/r11618 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph57.py/r90 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
compat-afdbot.py/r10249 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.0.2 Darwin/14.0.0 en-us -
 image/..
 text/..
AdMedia bot - -
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikipedia-radeh.py/r10269 Pywikipediabot/1.0 - -
 application/json
 -
add_category/r2179 Pywikipediabot/2.0 - -
 application/json
article_category_same_link/r2179 Pywikipediabot/2.0 - -
 application/json
pywiki-newar25.py/r10380 Pywikipediabot/1.0 - -
 application/json
botjagwar-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
mrajedrez-category_redirect.py/r11321 Pywikipediabot/1.0 - -
 text/..
 application/json
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
pywikipedia-git-wdph58.py/r90 Pywikipediabot/1.0 - -
 application/json
compat-commonscat.py/r10385 Pywikipediabot/1.0 - -
 application/json
compat-basictemplatedisamblist.py/r10331 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzredirectyeh.py/r10269 Pywikipediabot/1.0 - -
 application/json
 text/..
 application/xml
www.monit24.pl-m24Bot/4.1- - -
 image/..
 text/..
pywikipedia-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
compat-basictemplatedisamblist.py/r10385 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.54; ) - -
 text/..
featured/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
internationalization/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-interwiki.py/r10250 Pywikipediabot/1.0 - -
 application/xml
 application/json
compat-klbot_magiccat3.py/r10334 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
harvest_books/r2160 Pywikipediabot/2.0 - -
 application/json
SurakWare MediaWiki Bot/1.0 - -
 text/..
 application/xml
pywikipedia-git-wdph66.py/r90 Pywikipediabot/1.0 - -
 application/json
gsa-crawler (Enterprise; T4-DZLR68QD2A26J; mail address , mail address ) - -
 text/..
 image/..
 -
pywiki-newar25.py/r10348 Pywikipediabot/1.0 - -
 application/json
HTMLParser/2.0 - -
 text/..
 image/..
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
harvest_frtaxo/r2067 Pywikipediabot/2.0 - -
 application/json
erfgoedbot-categorize_images.py/r11633 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-pol.py/r10334 Pywikipediabot/1.0 - -
 application/json
compat-newinterwiki-fa.py/r10334 Pywikipediabot/1.0 - -
 application/json
Casual Web Crawler pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3;charset=UTF-8 -
 text/..
compat-radeh.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-aaAddDesc.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wd_move.py/r11775 Pywikipediabot/1.0 - -
 application/json
wpbot-blockpageschecker.py/r-1 (unknown) Pywikipediabot/1.0 - -
 text/..
 application/json
compat-zz_make_cat.py/r10269 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph59.py/r90 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en,* -
 image/..
 text/..
 application/json
compat-klbot_magiccat77.py/r10334 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/xml
 application/json
 application/rsd+xml
Mozilla/5.0 (compatible; Operabot/2.1)/Nutch-2.1 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia-milhist.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
pywikipedia-git-wdph65.py/r90 Pywikipediabot/1.0 - -
 application/json
 application/xml
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
aleph-avbot.py/r11368 Pywikipediabot/1.0 - -
 application/json
 text/..
compat-newinterwiki-ckb.py/r10334 Pywikipediabot/1.0 - -
 application/json
pywikipedia-soccer.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
pywikibot-compat-aaa_elenco.py/r10250 Pywikipediabot/1.0 - -
 application/json
pwb-klbot_web_en.py/r7 Pywikipediabot/1.0 - -
 application/json
createHierarchy/r2067 Pywikipediabot/2.0 - -
 application/json
voverb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
wAPI/1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
cwikt-interwiki.py/r10334 Pywikipediabot/1.0 - -
 application/json
pywikipedia-com_mis_license_t.py/r10261 Pywikipediabot/1.0 - -
 application/json
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Perl/User:Wdwdbot - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
User-Agent Baiduspider - -
 text/..
 image/..
 -
Optimus5 Spider - -
 text/..
Wiki.java 0.28 r153 (OctraBot 3.0) - -
 text/..
 application/json
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zztakhasosi.py/r10269 Pywikipediabot/1.0 - -
 application/json
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
diacritics_redirects/r2005 Pywikipediabot/2.0 - -
 application/json
compat-featured.py/r10360 Pywikipediabot/1.0 - -
 application/json
cwikt-interwiki.py/r10349 Pywikipediabot/1.0 - -
 application/json
Py-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (compatible; MyBot/1.0;) - -
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
moje-wymowa.py/r11775 Pywikipediabot/1.0 - -
 application/json
WPBot 1.0 - -
 text/..
 image/..
pywikipedia-checknewarticles_formaintenance-20130923.py/r11775 Pywikipediabot/1.0 - -
 application/json
pwb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
lmi_shortcuts/r2005 Pywikipediabot/2.0 - -
 application/json
pywikipedia-replace.py/r11747 Pywikipediabot/1.0 - -
 text/..
 application/json
unknown, 2a00:4b00:13d:200::129a Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
py-linktranslator.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph663.py/r90 Pywikipediabot/1.0 - -
 application/json
compat-Redlink.py/r10334 Pywikipediabot/1.0 - -
 application/json
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - -
 application/json
bot-VM-auto-erl.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
wikidata-coords_ghel.py/r10301 Pywikipediabot/1.0 - -
 application/json
compat-basicarticledisamblist.py/r10331 Pywikipediabot/1.0 - -
 application/json
py-newar25.py/r10375 Pywikipediabot/1.0 - -
 application/json
NikoBot1.0 - -
 text/..
 image/..
unknown, 2a00:4b00:13d:200::129c Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
unknown, 2a00:4b00:13d:200::129d Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.52; ) - -
 text/..
unknown, 2a00:4b00:13d:200::129e Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
harvest_template/r2067 Pywikipediabot/2.0 - -
 application/json
bot: fr-anal - -
 application/json
unknown, 2a00:4b00:13d:200::129b Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
wikiparser/1 CFNetwork/596.5 Darwin/12.5.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
3005485/r11657 Pywikipediabot/2.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129f Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
DeletionBot-deletion.py/r10244 Pywikipediabot/1.0 - -
 application/json
 application/xml
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 text/..
pywikipedia-wikidata_setproperty.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-rfubot.py/r10249 Pywikipediabot/1.0 - -
 application/json
load_category/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
py-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Optimus5 Spider en-us,en;q=0.5 -
 text/..
dmashine-CleanupTematic.py/r10334 Pywikipediabot/1.0 - -
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 - -
 application/json
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
maj_articles_recents/r2217 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-rdel2.py/r123 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
pywikipedia-zzhayati.py/r10269 Pywikipediabot/1.0 - -
 application/json
2896097/r11657 Pywikipediabot/2.0 - -
 application/json
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Zing-BottaBot/2.0 - -
 text/..
echooooooooooooo/r2090 Pywikipediabot/2.0 - -
 application/json
cwikt-interwiki.py/r10360 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph662.py/r90 Pywikipediabot/1.0 - -
 application/json
COIBot/2.0 - -
 text/..
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
botjagwar-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
botjagwar-getpron.py/r11215 Pywikipediabot/1.0 - -
 application/json
JavaCrawler/1.1 - -
 text/..
pywiki-cfd.py/r10360 Pywikipediabot/1.0 - -
 application/json
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
pywikipedia-git-wdph61.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-importwdcat.py/r90 Pywikipediabot/1.0 - -
 application/json
pywiki-yeke.py/r10348 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph65.py/r120 Pywikipediabot/1.0 - -
 application/json
 application/xml
parse_monument_article/r2005 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; UnisterBot; mail address ) - -
 text/..
theWxitBot/0.1 - -
 application/json
 image/..
compat-featured.py/r10354 Pywikipediabot/1.0 - -
 application/json
pwb-klbot_ncbi_fr.py/r7 Pywikipediabot/1.0 - -
 application/json
compat-zzinter.py/r10269 Pywikipediabot/1.0 - -
 application/json
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - -
 application/xml
 text/..
Wikibot/2.1.1 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 text/..
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
pwb-wikidata_time2ba.py/r7 Pywikipediabot/1.0 - -
 application/json
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
harvest_template/r2184 Pywikipediabot/2.0 - -
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
 -
pywikibot-compat-aaa_categorieAutori.py/r10250 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4 GoogleBot/2.1 - -
 image/..
 text/..
compat-featured.py/r10350 Pywikipediabot/1.0 - -
 application/json
chrwikt/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-featured.py/r10338 Pywikipediabot/1.0 - -
 application/json
COIBot/1.00 - -
 text/..
pywikipedia-wikidata_setproperty_NL_frominfobox_gemeentelink.py/r11200 Pywikipediabot/1.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
GoogleBot - -
 text/..
 image/..
pywikipedia-fr.wikt.RemiseEnForme.py/r68 Pywikipediabot/1.0 - -
 application/json
Crawler/1.2 - -
 text/..
compat-featured.py/r10329 Pywikipediabot/1.0 - -
 application/json
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 text/..
 -
 image/..
compat-basicarticledisamblist.py/r10376 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10334 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-cdel.py/r120 Pywikipediabot/1.0 - -
 application/json
User-Agent Baiduspider - -
 text/..
 image/..
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
Junut ping bot - -
 text/..
 -
cwikt-interwiki.py/r10385 Pywikipediabot/1.0 - -
 application/json
 application/xml
artists/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywiki-newar5.py/r10348 Pywikipediabot/1.0 - -
 application/json
pywikipedia-category_redirect.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
pywikipedia-git-wdph661.py/r90 Pywikipediabot/1.0 - -
 application/json
cwikt-interwiki.py/r10331 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-git-wdph61.py/r123 Pywikipediabot/1.0 - -
 application/json
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
GoogleBot-Image/1.0 bg -
 image/..
 text/..
wikiscore-MakeScoreTable.py/r11775 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
article_category_same_link/r2219 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
pywikipedia-refresh_merk.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
bitlybot - -
 text/..
 image/..
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 en-us -
 image/..
 text/..
 -
drp_warn/r2160 Pywikipediabot/2.0 - -
 application/json
YBot/0.1 - -
 application/vnd.php.serialized
ContextAd Bot 1.0 - -
 text/..
 image/..
 -
pywikipedia-com_wrong_info.py/r10261 Pywikipediabot/1.0 - -
 application/json
 text/..
rg/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
py-new_interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
job_scripts-2915873/r10301 Pywikipediabot/1.0 - -
 application/json
pywikipedia-pwb.py/r10348 Pywikipediabot/1.0 - -
 application/json
 text/..
 application/xml
 image/..
 application/ogg
python-wikitools/1.2 (User:LaraBot) - -
 application/json
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-wikifyer.py/r10360 Pywikipediabot/1.0 - -
 application/json
pwb/r2212 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
pywikipedia-interwiki.py/r161 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikifyer.py/r10376 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider; zh-CN -
 text/..
Wiki.java 0.27 r137 (OctraBot 2.9) - -
 text/..
 application/json
DotNetWikiBot/2.102 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
XLinkBot/1.00 - -
 text/..
save_from_commons/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
claimit/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-pwb.py/r10248 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11385 Pywikipediabot/1.0 - -
 application/json
 text/..
Mozilla/5.0 (compatible; SEO-crawler/1.0;) - -
 text/..
 audio/webm
Projet Sport-Projetsportliste2.py/r10533 Pywikipediabot/1.0 - -
 application/json
bot-rutr-accents.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
claimit/r2061 Pywikipediabot/2.0 - -
 application/json
HTMLParser/1.6 - -
 text/..
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
python-wikitools/1.2 (User:Mr.Z-bot) - -
 application/json
replace/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia4-nedavne.py/r11781 Pywikipediabot/1.0 - -
 text/..
 application/json
pywikipedia-nedavne.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
Inlibris.com XMLBot/1.0 - -
 text/..
Empedia Bot - -
 text/..
opentask-opentasks.py/r10326 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-importwdwisw.py/r123 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Spiderlytics/1.0; mail address ) de-de,de,en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
"echocrawl 2.0" - -
 text/..
 application/ogg
 -
 application/xml
Screaming Frog SEO Spider/2.20 - -
 text/..
 image/..
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
compat-basicarticledisamblist.py/r10377 Pywikipediabot/1.0 - -
 application/json
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia-imagerecat.py/r11775 Pywikipediabot/1.0 - -
 application/json
pwb-kimportlist.py/r7 Pywikipediabot/1.0 - -
 application/json
commons-substituter.py/r11436 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; ) - -
 text/..
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
CheMoBot/1.00 - -
 text/..
HESPERIAN_uploadculprits/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
weave_rest_bot, contact mail address - -
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
pywikipedia-featured.py/r10295 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-basicdisambcats.py/r10331 Pywikipediabot/1.0 - -
 application/json
msnbot - -
 text/..
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 -
 text/..
WikiBot/0.1 - -
 text/..
Bloom.fm Meta Crawler ( mail address ) - -
 text/..
23327.41total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Thu, Nov 21, 2013 22:25
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers