Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 May 2012 - 31 May 2012 (last 12 months)
000 ⇒ k
 

 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 67,897,030 page requests (mime type text/html only!) per day are considered crawler requests, out of 478,729,940 external requests, which is 14.2%

Page requests for crawlers that specify a url in the agent string
Count
x 1000
Secondary domain
(~site) name
URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cloudcrawling)
 code.google.com/p/crawler4j/text/..crawler4j (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~senchaiosrc)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: myproxywx)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat)
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~wikigraph2)
 code.google.com/appengineimage/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bie99miracle)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor)
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki4)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~deutiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nation4india)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vi-mobile)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4)
 code.google.com/p/rondaapplication/jsonRonda - url
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp6)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-data)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline0)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 114proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9)
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy)
 code.google.com/appengine-AppEngine-Google; (url; appid: s~senchaiosrc)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: maltingsproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 code.google.com/appenginetext/..Mozilla/5.0 AppEngine-Google; (url; appid: s~app3123ak)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp8)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kyaysarlay)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: betafxserver)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pox)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps002)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tmobile-internet)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yuricamara)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools6)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crowdsurfer100)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jptaravellahighschool)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nhsportal)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nothing-unusual)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: misc-tools)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wagagate)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ieultimateproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki1)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davidgotmoney50)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openeyeproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: predictionwrong)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ageryder)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sony-words)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kaveriselvaraj)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sizzsurf)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mboharsik)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web4proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivankrisproxyserver)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kuryproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: burmansearch)
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: downarchivestuffproxyserver)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cachehew)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~link123451)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-5)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-mastered)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn03)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kerouanen)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: laaplicaciondelucas)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tyangbanga)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp8)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: prfleme)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq0)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline8)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: seiyukyouen)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2)
 www.google.com/bot.htmlNONE/wikipedia- Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: vipoembed)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: djshan45)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azzaziprxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tinkernutsearch)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack)
 code.google.com/appenginetext/..Wiki.java 0.25 AppEngine-Google; (url; appid: wikipediatools)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: trabserver)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: futureducation1)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-phpproxy)
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 developers.facebook.comimage/..facebookplatform/1.0 (url)
 developers.facebook.com-facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
 developers.facebook.comtext/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.phpapplication/vnd.php.serializedfacebookexternalhit/1.1 (url)
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmapplication/vnd.php.serializedMozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html))
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: images-jpg)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via babelfish.yahoo.com)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible;bingbot/2.0;url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: p8roxy)
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: misc-tools)
 www.baidu.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htm-Baiduspider-image(url)
 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
 yandex.com/botsapplication/vnd.php.serializedMozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexNewslinks; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
 corp.naver.jp/text/..Mozilla/5.0 (compatible; NaverJapan/1.0; url)
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html-'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Nano; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
 help.yahoo.comtext/..Mozilla/5.0 (YahooYSMcm/3.0.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.html-Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
msn
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url)
 search.msn.com/msnbot.htm-msnbot/2.0b (url)
 search.msn.com/msnbot.htmapplication/vnd.php.serializedmsnbot-media/1.1 (url)
 search.msn.com/msnbot.htm-msnbot/0.01 (url)
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/-SeznamBot/3.0 (url)
cibra
 cibra.de/text/..CiBra Data Collector (url)
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
 ahrefs.com/robot/application/oggMozilla/5.0 (compatible; AhrefsBot/3.0; url)
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
 pear.php.net/text/..PEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8
 pear.php.net/image/..PEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.14
 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url)
 www.sogou.com/docs/help/webmasters.htm#07application/vnd.php.serializedSogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url)
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
wordpress
 klausgauger.wordpress.comtext/..WordPress/3.4-beta4-20825; url
 klima47.wordpress.comtext/..WordPress/3.4-beta4-20725; url
 kingcrimsonprog.wordpress.comimage/..WordPress/3.4-beta4-20766; url
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
 en.wikipedia.org/wiki/User_talk:Blevintrontext/..BlevintronBot version 2012-05-19 contact url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url)
 en.wikipedia.org/wiki/User_talk:Blevintrontext/..BlevintronBot version 2012-05-16 contact url
 en.wikipedia.org/wiki/User_talk:Blevintrontext/..BlevintronBot version 2012-05-09 contact url
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
 sk.wikipedia.org/wiki/Redaktor:TeslaBot-TeslaBot (url)
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.18.0 url
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.2 url
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19 url
soso
 help.soso.com/webspider.htmtext/..Sosospider(url)
 help.soso.com/webspider.htm-Sosospider(url)
discoveryengine
 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.html-Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.htmlimage/..Mozilla/5.0 (compatible; discobot/2.0; url)
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.38-15-generic; java 1.6.0_22; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0; java 1.7.0_03; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-desktop; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.42.12-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.9; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.18-238.19.1.el5xen; java 1.6.0_22; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.30-vs2.3.2.3-dq67sw; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0; java 1.7.0_03; Europe/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Indian/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.31-vs2.3.2.3-dq67sw; java 1.6.0_24; Europe/en) url
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/image/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 toolbar.youdao.com/image/..Youdao Toolbar (url)
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url)
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url)
zeebox
 www.zeebox.comapplication/jsonZeebox (url)
dataparksearch
 dataparksearch.org/bottext/..DataparkSearch/4.54-26052011 (url)
 dataparksearch.org/bottext/..DataparkSearch/4.54-2012-04-08 (url)
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
 toolserver.org/~dispenser/text/..DispensersTools (url)
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02
 toolserver.org/~dispenser/application/jsonDispensersTools (url)
yioop
 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url)
 www.yioop.com/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url)
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url)
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
wikidict
 www.wikidict.detext/..url
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url)
 www.gnip.com/image/..UnwindFetchor/1.0 (url)
 www.gnip.com/-UnwindFetchor/1.0 (url)
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url)
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
 www.archive.orgimage/..Mozilla/5.0 (compatible; heritrix/3.1.0 url)
 www.archive.orgtext/..Mozilla/5.0 (compatible; heritrix/3.1.0 url)
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)
sf
 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
 magpierss.sf.nettext/..MagpieRSS/0.72 (url; No cache)
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
commoncrawl
 www.commoncrawl.org/bot.htmltext/..CCBot/1.0 (url)
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url)
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url)
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.html-Mozilla/5.0 (compatible; bnf.fr_bot; url)
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url)
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
ephorus
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.3.0; url)
neofonie
 spider.neofonie.detext/..MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to mail address ; url; mail address )
 spider.neofonie.detext/..mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to mail address ; url; mail address )
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
 enwp.org/User:KingpinBottext/..KingpinBot (url)
 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
avantbrowser
 www.avantbrowser.comtext/..Avant Browser (url)
 www.avantbrowser.comtext/..Advanced Browser (url)
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
 github.com/NeilCrosby/wikislurpapplication/vnd.php.serializedWikiSlurp (url)
 github.com/edsu/wikitweetsapplication/jsonwikitweets <url
kalooga
 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url)
 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url)
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
butterflyfacts
 www.butterflyfacts.nettext/..WordPress/3.3.2; url
 www.butterflyfacts.nettext/..WordPress/3.1.2; url
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
feedshow
 www.feedshow.comtext/..FeedshowOnline (url)
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
newsgator
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
freebase
 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url)
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B206f); url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1 (Build 9B179b); url)
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url)
entireweb
 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url)
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url)
edu:8080
 vancouver.cs.washington.edu:8080/text/..Mozilla/5.0/heritrix/3.1.0 (compatible;; url)
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
kr:6600
 www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsptext/..url
whatrhymeswith
 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
wikiglass
 wikiglass.comtext/..url : mail address
tiscali
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.01.18 url)
SearchNearMe
 SearchNearMe.com/contact.phptext/..SearchNearMe (url)
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
orcabrowser
 www.orcabrowser.comtext/..Orca Browser (url)
it-influentials
 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
graemef
 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
feeds4all
 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
zootycoon
 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
snarfware
 www.snarfware.com/text/..Snarfer/0.x.x (url)
rssbandit
 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
superfeedr
 superfeedr.comapplication/xmlSuperfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
nemui
 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
seebot
 seebot.orgtext/..Lynx/2.8 (;url)
kula
 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
ponderer
 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/3.3 (Mac OS X; url; gzip-happy)
timewe
 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
zipcommander
 www.zipcommander.com/text/..1st ZipCommander (Net) - url
drupal
 drupal.org/text/..User-Agent: Drupal (url)
 drupal.org/text/..Drupal (url)
weblio
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
proximic
 www.proximic.comtext/..Mozilla/5.0 (compatible; proximic; url)
blogbridge
 www.blogbridge.com/text/..BlogBridge 2.13 (url)
winpodder
 winpodder.comtext/..WinPodder (url)
rssreader
 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
ranchero
 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
plagger
 plagger.org/text/..Plagger/0.x.xx (url)
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url)
spotinfluence
 spotinfluence.comtext/..spotinfluence/Nutch-1.4 (Spot Influence crawler; url; mail address )
 spotinfluence.com-spotinfluence/Nutch-1.4 (Spot Influence crawler; url; mail address )
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address )
cognarius
 cognarius.comapplication/jsonAppsArlak/1.0 (url)
embed
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url)
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url)
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address )
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url)
 duckduckgo.com/duckduckpreview.html-DuckDuckPreview/1.0; (url)
 duckduckgo.com/duckduckpreview.htmltext/..DuckDuckPreview/1.0; (url)
ac
 cse.iitkgp.ac.in/~rprtext/..Rajendra/Nutch-1.4 (Researcher; url; mail address )
 www.clips.ua.ac.be/pages/patternapplication/jsonPattern/2.3 url
netvibes
 www.netvibes.comtext/..Netvibes (url)
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
grid-son
 grid-son.comapplication/jsonurl
newmedialab
 labs.newmedialab.at/iSignals/text/..LMF 2.0rc6-SNAPSHOT/LinkedDataClientService running at url
govid
 govid.mobi/bot.phptext/..Mozilla/5.0 (compatible; gofind; url)
linkbutler
 www.linkbutler.de/spidertext/..lb-spider/Mozilla/5.0 Gecko/20100101 Firefox/10.0.2 (lb-spider; url; mail address )
Anonymouse
 Anonymouse.org/image/..url (Unix)
 Anonymouse.org/text/..url (Unix)
whstour
 whstour.com/osakatext/..WordPress/3.3.1; url
 whstour.com/tokyotext/..WordPress/3.3.1; url
 whstour.com/nagoyatext/..WordPress/3.3.1; url
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
genevasearch
 www.genevasearch.com/text/..gva/europe (Geneva Web Explorer.; url; mail address )
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
zapbot
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url)
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url)
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url)
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
 www.sonyericsson.com/UAprof/R800xR301.xmltext/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
pinterest
 pinterest.com/image/..Pinterest/0.1 url
froute
 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
thearchangelmichael
 ref.thearchangelmichael.nettext/..WordPress/3.3.1; url
 thearchangelmichael.nettext/..WordPress/3.3.1; url
ibis
 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
edu
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url)
suggy
 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
102392.059999993total

Page requests for probable crawlers, recognized by keyword
Count
x 1000
Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0
 application/json
 application/xml
 text/..
 -
 application/x-www-form-urlencoded
 image/..
php wikibot classes
 application/vnd.php.serialized
 text/..
GoogleBot-Image/1.0
 text/..
 image/..
 -
MediaWikiCrawler-Google/2.0 ( mail address )
 text/..
 -
MoovidaBot/0.1
 text/..
LinkParser/2.0
 text/..
 -
gsa-crawler (Enterprise; T2-DS3YYS6PYJWAS; mail address )
 text/..
 -
 image/..
 application/pdf
Peachy MediaWiki Bot API Version 1.0
 application/vnd.php.serialized
 text/..
GoogleBot-Image/1.0
 text/..
 image/..
 -
 application/vnd.php.serialized
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
 text/..
 -
wikiwix-bot-3.0
 text/..
 -
spider
 text/..
 application/vnd.php.serialized
 application/json
 image/..
Pywikipediabot/2.0
 application/json
 text/..
Answersbot
 text/..
ClueBot/1.1
 application/vnd.php.serialized
 text/..
ClueBot/2.0
 application/vnd.php.serialized
 text/..
Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
 text/..
MediaWiki::Bot/1.00
 text/..
 application/json
 -
BritannicaProjBot mail address
 text/..
wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
Mozilla 5.0 (Apibot 0.32)
 application/vnd.php.serialized
python-wikitools/1.2 (User:BernsteinBot)
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
 text/..
 image/..
 -
DigitalsmithsBot
 text/..
DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 image/..
 application/ogg
mail address
 application/vnd.php.serialized
 application/json
 text/..
 -
MediaWiki::Bot/3.2.6
 application/json
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
 application/x-www-form-urlencoded
 application/json
Spider website 0.2
 text/..
FAST Enterprise Crawler/5.3.4 ( mail address )
 text/..
 -
 image/..
 application/rsd+xml
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
 application/json
 text/..
GoogleBot 2.1
 text/..
cis455crawler
 text/..
 -
GSLFbot
 text/..
 image/..
 application/xml
DotNetWikiBot/2.100 (Unix 2.6.32.38; )
 text/..
 application/xml
plantspedia data crawler
 text/..
Tawbot (public svn release; plwiki)
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
 image/..
 text/..
 application/json
 -
MLBot (www.metadatalabs.com/mlbot)
 text/..
 application/vnd.php.serialized
 image/..
SineBot/1.5.18(User:SineBot)
 application/vnd.php.serialized
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/x-www-form-urlencoded
 application/xml
 -
Wikibot/1.57 CFNetwork/520.4.3 Darwin/11.4.0 (x86_64) (MacBookPro8,2)
 image/..
 text/..
 application/json
cleaner-wikipedia bot / self.maluke.com
 text/..
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 -
AnomieBOT 1.0 (TagDater)
 application/json
 text/..
Bot work. [[no:User:PladaskBot]].
 application/vnd.php.serialized
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
 application/json
Test Webbot
 text/..
 -
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
 text/..
 application/opensearchdescription+xml
Webwiki Search Engine Bot - www.webwiki.de
 text/..
php WalkingSoulBot
 application/vnd.php.serialized
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
 text/..
UCMore Crawler App
 text/..
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
 text/..
DotNetWikiBot/2.100 (Unix 5.10.0.0; )
 text/..
 application/xml
CorenSearchBot/1.5 en libwww-perl/6.02
 text/..
AniBot/0.9 php/curl
 application/vnd.php.serialized
AnomieBOT 1.0 (OrphanReferenceFixer)
 application/json
LinksCrawler 0.1beta
 text/..
 -
 image/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
 application/ogg
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 -
SiocWikiBot/1.0
 application/vnd.php.serialized
 text/..
HTMLParser/1.6
 text/..
 application/x-wiki
 image/..
SchoolReviewNetworkWikiBot
 application/json
wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
GermCrawler
 application/json
 text/..
Twitterbot/1.0
 text/..
 image/..
 -
JavaCrawler/1.1
 text/..
COIBot/1.00
 text/..
FAST Enterprise Crawler 6 used by Wipro Ltd ( mail address )
 text/..
 -
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
 application/json
SurakWare MediaWiki Bot/1.0
 text/..
 application/xml
DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
TrueKnowledgeBot bot mail address >
 application/vnd.php.serialized
 application/xml
AdMedia bot
 text/..
TVersity Media Robot
 text/..
HTMLParser/2.0
 text/..
 image/..
OrlodrimBot/1.0
 text/..
XLinkBot/1.00
 text/..
GttBot/0.3
 application/json
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
 application/json
 text/..
HosiryuhosiBot IRC-RecentChanges Util
 -
 text/..
bitlybot
 text/..
 image/..
 -
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
 application/json
super cool bot
 application/vnd.php.serialized
SiteSeekerCrawler/1.0
 text/..
HRoestBot, de-wikipedia using pywikipedia framework
 text/..
 application/json
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
 image/..
 text/..
 -
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
MyCuteBot/0.1
 text/..
 application/json
 application/vnd.php.serialized
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
 application/json
 text/..
AnomieBOT 1.0 (PERTableUpdater)
 application/json
 text/..
Mozilla/4.0 (compatible; merlinkbot/1.0)
 text/..
uw_transparancy, toolserver.org wikiproject parser
 application/json
AnomieBOT 1.0 (FlagIconRemover)
 application/json
Applied-Technologies-Inc-Spider/Nutch-1.4
 text/..
Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; mail address )
 text/..
 image/..
YBot/0.1
 application/vnd.php.serialized
OrangeCrawler/Nutch-1.0 ( mail address )
 text/..
wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
Tweakker crawler/Nutch-1.4
 text/..
typedef web-crawler ( mail address ) for Yandex Data Analysis School
 text/..
TheKeens bot
 text/..
AnomieBOT 1.0 (TemplateSubster)
 application/json
DotNetWikiBot/2.100 (Unix 3.0.0.12; )
 text/..
 application/xml
COIBot/2.0
 text/..
SearchBot
 text/..
FAST Enterprise Crawler 6 used by LexisNexis ( mail address )
 text/..
 -
 image/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.0/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
 application/xml
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.4.152;) Opera Mini/3.1
 image/..
 text/..
 -
Goalkeeperbot(User:Beetstra)/1.0
 text/..
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 368 subscribers; feed-id=3852576738117026533)
 application/xml
 -
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.1.0/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
python-wikitools/1.2 (User:LaraBot)
 application/json
ClueBot/2.0 (ClueBot NG Report Interface)
 text/..
My Nutch Spider/Nutch-1.4
 text/..
AnomieBOT 1.0 (BAGBot)
 application/json
 text/..
HBC Archive Indexerbot 0.9a
 text/..
MediaWiki::Bot 3.1.5
 application/json
Baiduspider
 text/..
WikiBot/0.1
 text/..
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
 application/xml
 text/..
Mozilla 5.0 (Apibot 0.30b5)
 application/vnd.php.serialized
Freebase Deathbot
 text/..
EarwigBot/0.1.dev.gitbc9fdf28 (Python/2.7.1; https://github.com/earwig/earwigbot; mail address )
 application/json
 application/x-www-form-urlencoded
Empedia Bot
 text/..
wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
GNAA-bot
 text/..
Mozilla/5.0 (Bgbot 0.5)
 text/..
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )"
 application/xml
 text/..
GoogleBot
 text/..
 image/..
FAST Enterprise Crawler/6.7.8 ( mail address )
 text/..
 -
Geni ircpybot 1.0
 text/..
 application/json
 application/xml
infraEnterprise v8 Web Crawler
 -
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; )
 text/..
UniversalFeedParser/5.1.1 https://code.google.com/p/feedparser/
 text/..
 application/xml
 -
MediaWiki::Bot/v3.4.2
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
 application/json
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
 text/..
 -
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.4.152;) Opera Mini/3.1
 image/..
 text/..
 -
OpenSearchServer_Bot
 text/..
My Bot
 text/..
 image/..
Shad robot/1.0
 text/..
MetallmanulBot for Wiktionary (run by Metallmanul)
 application/json
19527.15total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, Aug 10, 2012 12:10
Author:Erik Zachte (
Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers