Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Sep 2013 - 30 Sep 2013 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

Warning: all recent Wikimedia traffic analysis reports have been generated from old scripts.

The scripts are orphaned, and have not been maintained for at least 6 months. Many bugs are considerably older.
Known Bugzilla bugs: 46190, 46191, 46195, 46201, 46205, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 46289

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 132,807,630 page requests (mime type text/html only!) per day are considered crawler requests, out of 616,623,730 external requests, which is 21.5%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5 - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5 -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible;bingbot/2.0;url) - -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (X11; Linux x86_64; zh_CN) KHTML/4.10.5 Konqueror/4.10 AppEngine-Google; (url; appid: s~42overwall) zh-CN,en-US;q=0.9,en;q=0.8 -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~news-world-me) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: easypox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tom-server) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kaveriselvaraj) - -
 www.google.com/coop/cse/creftext/..PageFetcher-Google-CoOp;((url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: puthiyathiravidan) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz) - -
 code.google.com/appenginetext/..Acre/dev/53:918 factchecker.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 KHTML Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (url; appid: s~feedly-social) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: javahavens) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: this-is-not-what-u-think) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khrixy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: s~libmuteki2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lovelamp1988) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2; rv:23.0) Gecko/20100101 Firefox/23.0 AppEngine-Google; (url; appid: s~june0066y) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: maurandk-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icebre4ker) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: red-arg) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thepeterdeutsch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ipreflector2) - -
 code.google.com/appenginetext/..Acre/dev/59:906 ubiquity.freebaseapps.com AppEngine-Google; (url; appid: s~freebaseapps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cmd-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beholdonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mboharsik) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pxdrill) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ageryder) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sirpats) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: he4lproxy) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bypass-filter) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: randohnson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: matbuot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icecool1987) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: inetbrowse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mistakeproxyarea) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ucsm111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web4proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: funnyinternetwebsite) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitaniel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~liquid-helium) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyproxy2884) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cdeskinsp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: labnol-server-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jptaravellahighschool) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beansacks) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: raja584sekhar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kukucdoc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxynaungnaung) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ahsanovic) - -
 code.google.com/appenginetext/..Acre/dev/53:916 factchecker.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kauflog) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~stremor-crawler) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chilla-cell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demowaiy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: djsk-moon) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elliptical-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cyber-411) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~gg8mm8qq) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: longbows-hideout) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/27.0.1453.110 Safari/537.36 AppEngine-Google; (url; appid: s~livescorefeed-hrd) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hydraroxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vemmastar-c4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crowdsurfer100) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.0; url) en-us;q=0.7, en;q=0.3 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) es-es,es;q=0.8,en-us;q=0.5,en;q=0.3 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)/Nutch-2.1 en-us,en-gb,en;q=0.7,*;q=0.3 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: jpg8jpg) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh9) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.5b5 - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh4) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8gg6) en-US -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh10) en-US -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/1.1 (url) - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)1 readers - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/text/..WikipediaApp_svc/1.5.13 (url; mail address ) com.orange.mmp/1.7.0 - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/2.0; url - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/application/xmlAhrefsBot.Feeds v0.1; url - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url) - -
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/application/jsonYeti/1.1 (NHN Corp.; url) ja-JP,ja;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
news
 www.news.net/text/..News.Net/1.1 (url; mail address ) BasedOnRawPHP - -
 www.news.net/application/jsonNews.Net/1.1 (url; mail address ) BasedOnRawPHP - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
sblog
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test1; url) cs -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) - -
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/application/jsonMozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test2; url) * -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou News Spider/4.0(url) zh-cn -
wordpress
 josefboberg.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 y35pm.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 radishmag.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 managra.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 kingcrimsonprog.wordpress.comimage/..WordPress/3.7-alpha-25157; url - -
 klausgauger.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 at37.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 02varvara.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 tsjok45.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 dummidumbwit.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 drstvmovies.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 raymondpronk.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 sxoliastesxwrissynora.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 ahmadsamantho.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 josefboberg.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 radishmag.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 peripluscd.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 mediachecker.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
 halokue.wordpress.comtext/..WordPress/3.7-alpha-25157; url - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.27 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) -
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmapplication/jsonMozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmimage/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_35; Europe/de) url - -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.10-1-ARCH; java 1.7.0_40; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-30-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-1.16-desktop; java 1.7.0_40; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 2.6.32-50-server; java 1.6.0_27; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-29-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (superfi/global; amd64 Linux 2.6.32-042stab079.5; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-30-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-29-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; i386 Linux 3.2.0-37-generic-pae; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 3.4.4; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.4.4; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (superfi/global; amd64 Linux 2.6.32-042stab079.6; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_45; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86 Windows XP 5.1; java 1.7.0_25; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-29-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-30-generic; java 1.6.0_27; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-53-generic; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
bin-co
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
wikipedia
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
crossref
 alm.labs.crossref.orgapplication/jsonArticle Level Metrics - url - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url;) Gecko/2008032620 - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ru -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address es-es -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
scrapy
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.18.0 (url) kk -
 scrapy.orgimage/..Scrapy/0.18.2 (url) en -
 scrapy.orgtext/..Scrapy/0.18.2 (url) en -
 scrapy.orgtext/..Scrapy/0.18.0 (url) - -
 scrapy.orgtext/..Scrapy/0.18.1 (url) en -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 goo.gl/7y4SXimage/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url) - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.3 (url) -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/application/jsonDispensersTools (url) - -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
wikidict
 www.wikidict.detext/..url - -
zeerch
 zx1.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; ZXBOT-ZX2; url) - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
pchome
 www.pchome.com.tw/pchomebot.htmtext/..Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
profound
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20130905-2353 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.2-no_deferred_write-SNAPSHOT-20130910-1530 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics - url - -
 alm.plos.orgapplication/jsonPLOS Article Level Metrics - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics - url - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
 www.muso.comapplication/xmlMozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url) - -
 commons.wikimedia.org/wiki/User:Thumbnails_Check_Botimage/..Thumbnails_Check_Bot/0.1 (url; beta) - -
 tools.wikimedia.de/~para/GeoCommons/text/..url - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.97; url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
xbmc
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Linux; Debian GNU/Linux 7.0 (wheezy); 3.6.11 armv6l; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (iOS; 11.0.0, Version 5.1.1 (Build 9B830); url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1; url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) ja -
holasoyramon
 www.holasoyramon.com/blogtext/..WordPress/3.5.1; url - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) ja -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
 bsurprised.com/text/..BSurprised WikiBox 0.1 (url) en -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
rifanmuazin
 trends.rifanmuazin.comimage/..WordPress/3.6; url - -
 trends.rifanmuazin.comtext/..WordPress/3.6; url - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xml-url en,hi -
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en -
yunyun
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/SiteInfo.php?r=abouttext/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/SiteInfo.php?r=aboutimage/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
web
 www.web.nl/text/..web.nl spider/1.7; url - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
example
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 dev.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolTool/text/..MyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
import
 import.iotext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
ac
 www.roselab.sci.waseda.ac.jptext/..2QC (url; mail address ) - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 www.clips.ua.ac.be/patterntext/..Pattern/2.6 url - -
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
geolocation
 www.geolocation.wsapplication/jsonGeolocation.ws Wikimedia Commons import. url - -
linkdex
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.1; url) - -
github
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
 wummel.github.com/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.4; url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
trendiction
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
accelobot
 www.accelobot.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
sg-dev
 www.sg-dev.ch/application/jsonUser-Agent: SemioticSG/0.1 (url; mail address ) - -
kalooga
 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
 wiki.creativecommons.org/DiscoverEdtext/..My Nutch Spider/Nutch-1.7 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
psu
 citeseerx.ist.psu.edutext/..citeseerxbot (compatible; heritrix/1.14.4 url) - -
 citeseerx.ist.psu.eduimage/..citeseerxbot (compatible; heritrix/1.14.4 url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
diffbot
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
localhost
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
go
 kc.nict.go.jp/project1/crawl-ja.htmltext/..ICC-Crawler (Mozilla-compatible; mail address ; url) ja -
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
gulliway
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
su-jine
 www.su-jine.com/sujine_seo_textbrowser.phptext/..Su-Jine VirtualTextBrowser/0.03 (url) - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
metamagazine
 metamagazine.comtext/..WordPress/3.6; url - -
 metamagazine.comtext/..WordPress/3.5.2; url - -
uni-potsdam
 www.hpi.uni-potsdam.de/meinel/forschung/web_30/blog_intelligence.htmltext/..HPI-BI-Crawler/0.1(url)/Nutch-2.0-dev - -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
tivine
 tivine.com/application/jsonTivine_0.01 (url; mail address ) Test Developments - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) - -
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
 www.thinglink.com/help/ThinglinkImageBottext/..Elmer, the Thinglink ImageBot (url) - -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) - -
superfeedr
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.com-Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
rebelmouse
 rebelmouse.comimage/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.comtext/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url) - -
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
tvrage
 www.tvrage.com/application/vnd.php.serializedMyCoolTool/1.1 (url; mail address ) BasedOnOwnLib/1.1 - -
zapbot
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url) - -
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url) - -
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url) - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.9 (url) - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -
ibis
 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url) - -
 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
watchmouse
openwebspider
 www.openwebspider.org/text/..OpenWebSpider v0.1.4 (url) - -
archive-it
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
sf
 liferea.sf.net/application/xmlLiferea/1.8.3 (Linux; fr_FR.UTF-8; url) - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
kr:6600
 www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsptext/..url - -
yesup
 www.yesup.net/bot.htmltext/..Mozilla/5.0 (compatible; YesupBot/1.0; url) - -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
tiscali
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.10.2 url) * -
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.10.2 url) - -
nvdev
 qa.nvdev.comtext/..Netvibes (url) - -
sphider
 www.sphider.eu/about.phptext/..Sphider 1.3.6 url - -
158597.979999985total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 5) - -
 application/vnd.php.serialized
 text/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
 application/opensearchdescription+xml
LinkParser/2.0 - -
 text/..
 -
wikiwix-bot-3.0 - -
 text/..
 -
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 image/..
 -
 text/..
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
tigerbot - -
 application/json
 text/..
WikidataBot framework - -
 application/json
 text/..
GoogleBot-Image/1.0 - -
 text/..
 image/..
PythonWikipediaBot/1.0 - -
 application/json
 text/..
 application/xml
 -
pywikipedia-rad1.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
ClueBot/1.1 - -
 application/vnd.php.serialized
spider - -
 application/vnd.php.serialized
 text/..
 application/json
 image/..
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
 -
redirect/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
YisouSpider - -
 text/..
 -
 application/ogg
ClueBot/2.0 - -
 application/vnd.php.serialized
pywikipedia-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia2-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_descr.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 image/..
 video/webm
MediaWiki::Bot/5.005006 - -
 application/json
Semantix Bot 0.1 - -
 text/..
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
 application/json
DigitalsmithsBot - -
 text/..
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 - -
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
itemfix/r2046 Pywikipediabot/2.0 - -
 application/json
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Bot/3.2.6 - -
 application/json
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; SearchBot) - -
 text/..
www.integromedb.org/Crawler - -
 text/..
 -
 application/json
 application/xml
 image/..
pywikipedia-git-wdph57.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Tawbot (public svn release; plwiki) - -
 text/..
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
 application/ogg
 video/webm
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
moje-wymowa.py/r11483 Pywikipediabot/1.0 - -
 application/json
botjagwar-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
maj_articles_recents/r2033 Pywikipediabot/2.0 - -
 application/json
maj_articles_recents/r2061 Pywikipediabot/2.0 - -
 application/json
itemfix/r1912 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
 -
pywikipedia-wikidata_coordinaten_NL.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
compat-afdbot.py/r10249 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
 application/json
itemfix/r2053 Pywikipediabot/2.0 - -
 application/json
Keybot Translation-Search-Machine - -
 text/..
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
 image/..
 application/json
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
milog_bot/1.0 ( mail address ) - -
 text/..
maj_articles_recents/r2108 Pywikipediabot/2.0 - -
 application/json
360spider-image - -
 image/..
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
itemfix/r2052 Pywikipediabot/2.0 - -
 application/json
pywikipedia-fr.wikt.RemiseEnForme.py/r68 Pywikipediabot/1.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
GermCrawler - -
 application/json
 text/..
 application/xml
 application/ogg
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Test Webbot - -
 text/..
moje-porzucone.py/r11483 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 -
 application/opensearchdescription+xml
 application/rsd+xml
Bot: mail address - -
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
maj_articles_recents/r2015 Pywikipediabot/2.0 - -
 application/json
maj_articles_recents/r2050 Pywikipediabot/2.0 - -
 application/json
bot/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - -
 application/json
matilda/r2093 Pywikipediabot/2.0 - -
 application/json
COIBot/1.00 - -
 text/..
BOT/0.1 (BOT for JCE) - -
 text/..
 -
My Nutch Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
 application/pdf
pywikipedia-wikidata_cleanlabel.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; SearchBot) ru,en;q=0.1,*;q=0.01 -
 text/..
p-welcome-d-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 text/..
 application/json
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
harvest_template/r2049 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
claimit/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
bot-linkalt.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
maj_articles_recents/r2003 Pywikipediabot/2.0 - -
 application/json
pywikipedia3-MakeScoreTable_test2.py/r11448 Pywikipediabot/1.0 - -
 application/json
claimit/r2061 Pywikipediabot/2.0 - -
 application/json
wikidata-sitelinks.py/r11618 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.52; ) - -
 text/..
pyadmin-autoredirect.py/r11714 Pywikipediabot/1.0 - -
 application/json
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Wikibot/2.1.1 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 text/..
Python27-pythonw.exe/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10249 Pywikipediabot/1.0 - -
 application/json
 application/xml
botjagwar-anagrama.py/r11215 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
NutchCrawler/Nutch-2.2.1 - -
 text/..
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) - -
 application/json
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en,* -
 image/..
 text/..
 application/json
itemfix/r2051 Pywikipediabot/2.0 - -
 application/json
bot: fr-anal - -
 application/json
pywikipedia-git-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; wmbot; ) - -
 text/..
 image/..
anagrama/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Sogou web spider/4.0 zh-cn -
 text/..
 -
new2-zzdi1.py/r11452 Pywikipediabot/1.0 - -
 text/..
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
 application/xml
 application/rsd+xml
 application/opensearchdescription+xml
pywikipedia-wikidata_coordinaten_coor_title_dec.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
itemfix/r2049 Pywikipediabot/2.0 - -
 application/json
www.monit24.pl-m24Bot/4.1- - -
 image/..
 text/..
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
ko_missing/r11703 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-interwiki.py/r10250 Pywikipediabot/1.0 - -
 application/xml
 application/json
COIBot/2.0 - -
 text/..
pywikipedia-git-importwdwi.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
commons-imagerecat.py/r11028 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; UnisterBot; mail address ) - -
 text/..
 application/json
 image/..
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
HTMLParser/2.0 - -
 text/..
 image/..
pywikipedia-radeh.py/r10269 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph38.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
NikoBot1.0 - -
 text/..
 image/..
pywikipedia-replace.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
maj_articles_recents/r2098 Pywikipediabot/2.0 - -
 application/json
compat-featured.py/r10250 Pywikipediabot/1.0 - -
 application/json
 application/xml
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
XLinkBot/1.00 - -
 text/..
compat-checkimages.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
ht2p-bot/v1 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
wAPI/1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
itemfix/r2045 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph22.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-VM-auto-erl.py/r11775 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
 -
ws_crawl en -
 text/..
 application/xml
SurakWare MediaWiki Bot/1.0 - -
 text/..
Cyberfox Spider ru, en -
 text/..
 -
add_category/r2061 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
AdMedia bot - -
 text/..
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
ko_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
pywikipedia-archivebot.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/xml
 application/json
Xaldon WebSpider 2.7.b8 - -
 text/..
 -
pywikipedia-featured.py/r10280 Pywikipediabot/1.0 - -
 application/json
 application/xml
maj_articles_recents/r2031 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-aaa_elenco.py/r10250 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-featured.py/r33 Pywikipediabot/1.0 - -
 application/json
 application/xml
add_template/r2061 Pywikipediabot/2.0 - -
 application/json
JavaCrawler/1.1 - -
 text/..
pywikipedia-featured.py/r11775 Pywikipediabot/1.0 - -
 application/json
maj_articles_recents/r2032 Pywikipediabot/2.0 - -
 application/json
newser-wikinewser2.py/r11775 Pywikipediabot/1.0 - -
 application/json
MediaWiki Catalozhny Snake Robot/1.1 - -
 application/json
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
pywikipedia-pwb.py/r10248 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
 image/..
botjagwar-getpron.py/r11215 Pywikipediabot/1.0 - -
 application/json
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
pywikipedia-category_redirect.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
ParallelCorpusRobot/1.0 - -
 text/..
 application/xml
pywikipedia-aaAddDescQuery.py/r11775 Pywikipediabot/1.0 - -
 application/json
dabtouch/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-replace.py/r11747 Pywikipediabot/1.0 - -
 text/..
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
WPBot 1.0 - -
 text/..
erfgoedbot-categorize_images.py/r11633 Pywikipediabot/1.0 - -
 application/json
 application/xml
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.0.2 Darwin/14.0.0 en-us -
 image/..
 text/..
add_list/r2061 Pywikipediabot/2.0 - -
 application/json
GoogleBot en-us,en;q=0.5 -
 text/..
pywikipedia-refresh_merk.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
iw2data2/r11703 Pywikipediabot/2.0 - -
 application/json
commons-commons-commands.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-wd_move.py/r11775 Pywikipediabot/1.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129f Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
voverb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r10280 Pywikipediabot/1.0 - -
 application/json
2679279/r11657 Pywikipediabot/2.0 - -
 application/json
mrajedrez-category_redirect.py/r11321 Pywikipediabot/1.0 - -
 text/..
 application/json
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
run_replace.6r2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129a Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
wpbot-blockpageschecker.py/r-1 (unknown) Pywikipediabot/1.0 - -
 text/..
 application/json
unknown, 2a00:4b00:13d:200::129d Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
pywikipedia-zztakhasosi.py/r10269 Pywikipediabot/1.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129c Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
DeletionBot-deletion.py/r10244 Pywikipediabot/1.0 - -
 application/json
 application/xml
rg/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-zzinter.py/r10269 Pywikipediabot/1.0 - -
 application/json
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 - -
 application/json
botjagwar-voverb.py/r11215 Pywikipediabot/1.0 - -
 application/json
User-Agent Baiduspider - -
 text/..
 image/..
 -
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Math. Comp. PA Crawler - -
 text/..
maj_articles_recents/r1995 Pywikipediabot/2.0 - -
 application/json
run_replace.6r1/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
HTMLParser/1.6 - -
 text/..
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 image/..
 -
 application/ogg
artists/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129b Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
compat-checkimages.py/r10251 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzhayati.py/r10269 Pywikipediabot/1.0 - -
 application/json
GoogleBot - -
 text/..
run_replace.6r3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-featured.py/r10308 Pywikipediabot/1.0 - -
 application/json
 application/xml
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
AppCodes crawler - looking for iOS app mentions. More info: mail address Robots.txt id: AppCodesCrawler - -
 text/..
compat-featured.py/r10300 Pywikipediabot/1.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129e Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
theWxitBot/0.1 - -
 application/json
 image/..
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4 GoogleBot/2.1 - -
 image/..
 text/..
py-linktranslator.py/r11775 Pywikipediabot/1.0 - -
 application/json
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-findbox2.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
msnbot - -
 text/..
Zing-BottaBot/2.0 - -
 text/..
Inlibris.com XMLBot/1.0 - -
 text/..
cwikt-interwiki.py/r10245 Pywikipediabot/1.0 - -
 application/xml
 application/json
botjagwar-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
wikiparser/1 CFNetwork/596.4.3 Darwin/12.4.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
pywikipedia-wikidata_setproperty.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph61.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 - -
 application/json
FAST Search Web Crawler 14.0.0291.0000 - -
 text/..
 -
pywikipedia-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
g13bot_tools-g13_db_maintenance.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
pywikibot-compat-aaa_categorieAutori.py/r10250 Pywikipediabot/1.0 - -
 application/json
compat-checkimages.py/r10269 Pywikipediabot/1.0 - -
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
 -
aleph-avbot.py/r11368 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-redirect.py/r11385 Pywikipediabot/1.0 - -
 application/json
 text/..
article_to_category/r2061 Pywikipediabot/2.0 - -
 application/json
compat-category.py/r10249 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r10269 Pywikipediabot/1.0 - -
 application/json
pywikipedia-milhist.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
Wikipediabot-wikidata_gndcheck.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11674 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
Wiki.java 0.27 r137 (OctraBot 2.9) - -
 text/..
 application/json
pywikipedia-imagerecat.py/r11775 Pywikipediabot/1.0 - -
 application/json
featured/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
YBot/0.1 - -
 application/vnd.php.serialized
pywikipedia-hotarticle.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
people_dates/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-importwdwi.py/r6 Pywikipediabot/1.0 - -
 application/json
Projet Sport-Projetsportliste2.py/r10533 Pywikipediabot/1.0 - -
 application/json
 application/xml
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 text/..
 -
 image/..
Casual Web Crawler pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3;charset=UTF-8 -
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
g13bot_tools-nrsiref_list.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
GoogleBot/1.6 (Acer Core i3; Intel Linux Centos 6.4; en-US; rv:1.9.2.2) Java/1.7u3 Centos/6.4 - -
 text/..
pywikipedia-zzredirectyeh.py/r10269 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r10300 Pywikipediabot/1.0 - -
 application/json
bot-itnoun.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r10295 Pywikipediabot/1.0 - -
 application/json
pywikipedia-soccer.py/r11775 Pywikipediabot/1.0 - -
 text/..
 application/json
pywikipedia-git-cwi.py/r67 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r10308 Pywikipediabot/1.0 - -
 application/json
parse_monument_article/r2004 Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
Mozilla/5.0 (compatible; GufyBot) - -
 text/..
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.102 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
Empedia Bot - -
 text/..
harvest_template/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Jbot - -
 text/..
Bot - -
 text/..
compat-rfubot.py/r10249 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzlangcat.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
python-wikitools/1.2 (User:LaraBot) - -
 application/json
commons-substituter.py/r11436 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider; zh-CN -
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
cd-delinker.py/r11571 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
refresh_merk/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r10301 Pywikipediabot/1.0 - -
 application/json
Curious George - www.analyticsseo.com/crawler - -
 text/..
pywikipedia-aaAddDescArt.py/r11775 Pywikipediabot/1.0 - -
 application/json
FAST Search Web Crawler 14.0.0325.0000 - -
 text/..
 -
Mozilla/5.0 (compatible; wiki parser thing - -
 application/json
bitlybot - -
 text/..
 image/..
synchbot-__init__.py/r11775 Pywikipediabot/1.0 - -
 application/json
wikiscore-MakeScoreTable.py/r11549 Pywikipediabot/1.0 - -
 application/json
opentask-opentasks.py/r10326 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 text/..
 -
Crawler/1.2 - -
 text/..
replace/r2090 Pywikipediabot/2.0 - -
 application/json
"echocrawl 2.0" - -
 text/..
 application/ogg
 -
pywikipedia-wikidata_setproperty_NL_frominfobox_gemeentelink.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
Category bot/1.0 based on Python urllib2 mail address - -
 application/json
mijnbots-ZenV.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/xml
 application/json
khaCrawler - -
 text/..
Sogou Web Spider zh-cn -
 text/..
pywikipedia-wlmcommonscat.py/r11102 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r10307 Pywikipediabot/1.0 - -
 application/json
ZeBigWebBot - -
 text/..
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
createHierarchy/r2067 Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (CHUUClerk; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikipedia-redirect.py/r10261 Pywikipediabot/1.0 - -
 application/json
WikiBot/0.1 - -
 text/..
 image/..
MediaWiki::Bot/3.005002 - -
 application/json
g13bot_tools-g13_nom_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
IssueCrawler - -
 text/..
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
DotNetWikiBot/2.102 (Unix 3.2.0.52; ) - -
 text/..
bot-nlpart.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Mozilla/5.0 GoogleBot/2.1 (Linux; Android/4.3; Galaxy Nexus Build/JZO54K) AppleWebKit/537.36 KHTML Chrome/28.0.1500.71 Mobile Safari/537.36 - -
 text/..
harvest_template/r2090 Pywikipediabot/2.0 - -
 application/json
pywikipedia-zzporb-main.py/r10269 Pywikipediabot/1.0 - -
 application/json
WPendata-image_sur_enNaissance.py/r10533 Pywikipediabot/1.0 - -
 application/json
pywikipedia-com_wrong_info.py/r10261 Pywikipediabot/1.0 - -
 application/json
 text/..
MediaWiki::Bot 3.1.5 - -
 application/json
Screaming Frog SEO Spider/2.20 - -
 text/..
 image/..
pywikipedia-wlmfitxers.py/r11102 Pywikipediabot/1.0 - -
 application/json
24891.68total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Oct 7, 2013 16:00
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers