Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Feb 2012 - 29 Feb 2012 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 66,805,760 page requests (mime type text/html only!) per day are considered crawler requests, out of 477,009,310 external requests, which is 14.0%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~senchaiosrc)
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy)
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3)
 code.google.com/appengineapplication/jsonMozilla 4.0 AppEngine-Google; (url; appid: prfleme)
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; url)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: prfleme)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-phpproxy)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url)
 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~expinia-wiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: myproxywx)
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp6)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100thpriest)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kutchix)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thakurproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: good-proxy)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp8)
 code.google.com/p/rondaapplication/jsonRonda - url
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nmimsforti)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline7)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: prexytwo)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cmd-proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: djsk-moon)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aero-proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~willster273)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81)
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxynaungnaung)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn03)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: craigserver)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pox)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kevsproxy)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: buyitnw)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openeyeproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: prexyproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: argim-free)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hackzq8search)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gilithernil)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sixcareproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: no-restrict)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demowaiy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps005)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: inetbrowse)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyproxy2884)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: no9-bbs)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gangstertownusa)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hydraroxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: korvas-sux)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kyaysarlay)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aboytes13tls)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: captainfigolu)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kerouanen)
 www.google.com/bot.htmlNONE/wikipedia- Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elliptical-proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gvhiemenz)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxygeekcoke)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gizmo-jumpjet)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mhomeroxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: arturoproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps001)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zfqproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~no8lancelot)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ottogrib)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools6)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: suzetteklierocks)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tmobile-internet)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxworx)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: itechgiz-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: philipproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jptaravellahighschool)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beansacks)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tinkersearch)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: shadowtaker-proxy)
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog)
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: spenta-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: iamtheky)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fahad-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~b2-apps)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: z-prox)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wagagate)
 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: vipoembed)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: workrelatedworkstuff)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ipreflector2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan)
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 developers.facebook.comimage/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url)
 developers.facebook.comtext/..facebookplatform/1.0 (url)
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
 www.bing.com/bingbot.htmapplication/vnd.php.serializedMozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
 www.bing.com/bingbot.htmtext/..User-Agent :Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxy2fly0)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via babelfish.yahoo.com)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: provided-by)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: tcpudp10)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy0)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b4
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: updatedit)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyflyfly9)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: cgiproxy6)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: bereturning)
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) [UsableNet Lift Mobile]
 www.google.com/bot.htmltext/..Mozilla/5.0(compatible;GoogleBot/2.1;url)
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.comtext/..Mozilla/5.0 (YahooYSMcm/3.0.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-27.htmltext/..DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; url)
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsapplication/vnd.php.serializedMozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexNewslinks; url)
msn
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url)
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; archive.org_bot url)
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
 www.archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; archive.org_bot url)
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url)
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
 wwwgogetpapers.com/text/..User-Agent: GoGetPapersBot (url)
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
yacy
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-220.4.1.el6.i686; java 1.6.0_22; US/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_29; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 SunOS 5.11; java 1.6.0_26; US/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32.36-228-scalaxy; java 1.6.0_18; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_02; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.6-2-ARCH; java 1.7.0_03-icedtea; Asia/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-25-server; java 1.6.0_20; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.41.10-3.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-38-generic; java 1.6.0_20; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.18-028stab091.2; java 1.6.0_20; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.2-1.fc16.x86_64; java 1.7.0_b147-icedtea; W-SU/ru) url
 yacy.net/bot.htmltext/..yacybot (webportal/global; amd64 Linux 2.6.32-37-server; java 1.6.0_20; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.42.3-2.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_30; Europe/hr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/no) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_23; Indian/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.0.0-13-generic-pae; java 1.6.0_23; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.16.46-0.12-smp; java 1.6.0_15; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.7-1-ARCH; java 1.7.0_03-icedtea; Asia/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.7.0_147-icedtea; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-220.4.2.el6.i686; java 1.6.0_22; US/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-15-generic; java 1.6.0_26; Europe/sv) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_29; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; sparc SunOS 5.11; java 1.7.0; GMT01:00/en) url
 yacy.net/bot.htmltext/..yacybot (allip/any; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/ca) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.38-8-server; java 1.6.0_22; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.7.0_02; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.2-1.fc16.x86_64; java 1.7.0_b147-icedtea; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.24-26-generic; java 1.6.0_18; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.0.0-15-generic; java 1.6.0_23; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.5-gentoo; java 1.6.0_22; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Asia/zh) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_17; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; America/en) url
 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-39-generic; java 1.6.0_20; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.24-gentoo-r4; java 1.6.0_15; GMT/ru) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/es) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.5-1-ARCH; java 1.7.0_147-icedtea; Asia/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_27; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.5-3.fc16.x86_64; java 1.7.0_b147-icedtea; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.5-1-ARCH; java 1.6.0_24; US/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.6-3.fc16.x86_64; java 1.7.0_b147-icedtea; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_20; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-24-server; java 1.6.0_26; Asia/zh) url
 yacy.net/bot.htmltext/..yacybot (seofood/any; amd64 Linux 2.6.18-238.5.1.el5.028stab085.3; java 1.6.0_26; Europe/de) url
wordpress
 pennylibertygbow.wordpress.comtext/..WordPress/3.4-alpha-19904; url
 josefboberg.wordpress.comtext/..WordPress/3.4-alpha-19904; url
 driwancybermuseum.wordpress.comtext/..WordPress/3.4-alpha-19904; url
 driwancybermuseum.wordpress.comtext/..WordPress/3.4-alpha-19719; url
 eof737.wordpress.comtext/..WordPress/3.4-alpha-19904; url
 driwancybermuseum.wordpress.comtext/..WordPress/3.4-alpha-19814; url
 02varvara.wordpress.comtext/..WordPress/3.4-alpha-19904; url
 einflussreicheleute.wordpress.comtext/..WordPress/3.4-alpha-19904; url
 curtisnarimatsu.wordpress.comtext/..WordPress/3.4-alpha-19904; url
 greatriversofhope.wordpress.comtext/..WordPress/3.4-alpha-19904; url
 pennylibertygbow.wordpress.comtext/..WordPress/3.4-alpha-19827; url
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/-SeznamBot/3.0 (url)
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
 pear.php.net/text/..PEAR HTTP_Request class ( url )
 pear.php.net/image/..PEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2application/jsonHTTP_Request2/2.0.0 (url) PHP/5.3.8
 pear.php.net/package/http_request2text/..HTTP_Request2/2.0.0 (url) PHP/5.3.2-1ubuntu4.10
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.1; url)
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/image/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 toolbar.youdao.com/image/..Youdao Toolbar (url)
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ),gzip(gfe) (via translate.google.com)
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07application/vnd.php.serializedSogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou inst spider/4.0(url)
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
discoveryengine
 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/2.0; url)
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url)
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.18.0 url
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
 en.wikipedia.orgtext/..url
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19 url
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
archive-it
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url)
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
 www.semager.de/blog/semager-bots/-Mozilla/5.0 (compatible; Semager/1.4c; url)
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
 toolserver.org/~dispenser/text/..DispensersTools (url)
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02
 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)
soso
 help.soso.com/webspider.htmtext/..Sosospider(url)
 help.soso.com/webspider.htm-Sosospider(url)
 help.soso.com/webspider.htmapplication/xmlSosospider(url)
sf
 magpierss.sf.netapplication/xmlMagpieRSS/0.72 (url; No cache)
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
yioop
 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url)
 www.yioop.com/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url)
wikidict
 www.wikidict.detext/..url
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url)
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url)
commoncrawl
 www.commoncrawl.org/bot.htmltext/..CCBot/1.0 (url)
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url)
 help.goo.ne.jp/help/article/1142text/..ichiro/3.0 (url)
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
 help.goo.ne.jp/help/article/1142/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
 enwp.org/User:KingpinBottext/..KingpinBot (url)
 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
ac
 ce.yazduni.ac.irtext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 cse.iitkgp.ac.in/~rprtext/..Parnab/Nutch-0.9 (IIT Kharagpur; url; mail address )
 www.cse.iitb.ac.in/~vishaal_h4text/..Arjun/Nutch-0.9 (IIT Kharagpur; url; mail address )
 ce.yazduni.ac.ir-Mozilla/5.0 (compatible; heritrix/1.14.4 url)
daum
 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
 github.com/NeilCrosby/wikislurpapplication/vnd.php.serializedWikiSlurp (url)
 github.com/tenderlove/mechanize/text/..Mechanize/2.0.1 Ruby/1.9.2p290 (url)
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
newsgator
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
avantbrowser
 www.avantbrowser.comtext/..Avant Browser (url)
 www.avantbrowser.comtext/..Advanced Browser (url)
feedshow
 www.feedshow.comtext/..FeedshowOnline (url)
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
lipperhey
 www.lipperhey.com/text/..Mozilla/5.0 (compatible; Lipperhey Site Explorer; url)
whatrhymeswith
 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
artez
 www.artez.nltext/..artezTest/0.1 (url)
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address )
kr:6600
 www.checkprivacy.or.kr:6600/RS/PRIVACY_FAQ.jsptext/..url
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url)
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url)
suggy
 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
thearchangelmichael
 thearchangelmichael.nettext/..WordPress/3.3.1; url
 info.thearchangelmichael.nettext/..WordPress/3.3.1; url
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
 tinyurl.com/64t5napplication/xmlRome Client (url) Ver: UNKNOWN
freebase
 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
tkolb
 www.tkolb.de/rissbot.txttext/..RISSBot/dev (url)
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
whstour
 tokyo.whstour.comtext/..WordPress/3.2.1; url
 osaka.whstour.comtext/..WordPress/3.2.1; url
 nagoya.whstour.comtext/..WordPress/3.2.1; url
wikiglass
 wikiglass.comtext/..url : mail address
ranchero
 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
bne
 www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.htmltext/..Mozilla/5.0 (compatible; archive.org_bot/3.1.1 url)
 www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot/3.1.1 url)
timewe
 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
ponderer
 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
wattsupwiththat
 wattsupwiththat.comtext/..WordPress/3.4-alpha-19904; url
 wattsupwiththat.comtext/..WordPress/3.4-alpha-19719; url
 wattsupwiththat.comtext/..WordPress/3.4-alpha-19814; url
rssbandit
 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
nemui
 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
plagger
 plagger.org/text/..Plagger/0.x.xx (url)
zootycoon
 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
rssreader
 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
graemef
 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
orcabrowser
 www.orcabrowser.comtext/..Orca Browser (url)
winpodder
 winpodder.comtext/..WinPodder (url)
it-influentials
 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
zipcommander
 www.zipcommander.com/text/..1st ZipCommander (Net) - url
seebot
 seebot.orgtext/..Lynx/2.8 (;url)
blogbridge
 www.blogbridge.com/text/..BlogBridge 2.13 (url)
snarfware
 www.snarfware.com/text/..Snarfer/0.x.x (url)
kula
 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
 SearchNearMe.com/contact.phptext/..SearchNearMe (url)
feeds4all
 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address )
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
entireweb
 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
tumblr
 benderthewebrobot.tumblr.comtext/..Mozilla/5.0 (compatible; Bender; url)
 benderthewebrobot.tumblr.comapplication/vnd.php.serializedMozilla/5.0 (compatible; Bender; url)
edu:8080
 vancouver.cs.washington.edu:8080/text/..Mozilla/5.0/heritrix/3.1.0 (compatible;; url)
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
superfeedr
 superfeedr.comapplication/xmlSuperfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
 superfeedr.comtext/..Superfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/3.3 (Mac OS X; url; gzip-happy)
blogscope
 www.blogscope.net/text/..Mozilla/5.0 (compatible; BlogScope/1.0; url; U of Toronto)
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
vocationalschools
 vocationalschools.metext/..WordPress/3.3.1; url
weblio
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url)
edister
 www.edister.com/bot.htmltext/..EdisterBot (url)
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
scoutjet
 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
drupal
 drupal.org/text/..User-Agent: Drupal (url)
 drupal.org/text/..Drupal (url)
metamoji
 www.metamoji.com/jp/crawler.htmltext/..Mozilla/5.0 (compatible; MetamojiCrawler/1.0; url
fotopedia
 www.fotopedia.comapplication/jsonPicor (url)
froute
 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
netvibes
 www.netvibes.comtext/..Netvibes (url)
pinterest
 pinterest.com/image/..Pinterest/0.1 url
search
 www.search.ch/rim.htmltext/..UltraSpider3000/1.0 (url)
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
wesee
 www.wesee.com/en/support/bot/text/..WeSEE:Search/0.1 (Alpha, url)
 www.wesee.com/en/support/bot/image/..WeSEE:Search/0.1 (Alpha, url)
mondowindow
 www.mondowindow.comtext/..MondoWindow (url)
Anonymouse
 Anonymouse.org/image/..url (Unix)
 Anonymouse.org/text/..url (Unix)
infospress
 www.infospress.comtext/..WordPress/3.3.1; url
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url
semiocast
 semiocast.com/text/..Mozilla/5.0 (compatible; Semiocast HTTP client; url)
 semiocast.com/application/xmlMozilla/5.0 (compatible; Semiocast HTTP client; url)
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url)
cmu
 boston.lti.cs.cmu.edu/crawler_12/text/..Mozilla/5.0 (compatible; lemurwebcrawler mail address ; url)
 boston.lti.cs.cmu.edu/crawler_12/image/..Mozilla/5.0 (compatible; lemurwebcrawler mail address ; url)
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
ibis
 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
linkfluence
 linkfluence.net/text/..Mozilla/5.0 (compatible; linkfluence/0.9; url)
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url)
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url)
summify
 summify.comtext/..Summify (Summify/1.0.1; url)
z-add
 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address )
99544.349999996total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0
 application/json
 application/xml
 text/..
 image/..
 -
MediaWikiCrawler-Google/2.0 ( mail address )
 text/..
 -
GoogleBot-Image/1.0
 image/..
 text/..
 -
php wikibot classes
 application/vnd.php.serialized
 text/..
 -
LinkParser/2.0
 text/..
mail address
 application/vnd.php.serialized
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
 text/..
 -
ClueBot/1.1
 application/vnd.php.serialized
ClueBot/2.0
 application/vnd.php.serialized
 -
wikiwix-bot-3.0
 text/..
 -
Answersbot
 text/..
 -
GoogleBot-Image/1.0
 text/..
 image/..
 application/vnd.php.serialized
 -
 application/json
spider
 text/..
 application/json
 image/..
 application/vnd.php.serialized
 -
Pywikipediabot/2.0
 application/json
Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
 text/..
 application/vnd.php.serialized
 -
 image/..
 application/xml
Mozilla 5.0 (Apibot 0.32)
 application/vnd.php.serialized
 text/..
Onespot Crawler
 application/json
 text/..
 -
YBot/0.1
 application/vnd.php.serialized
DigitalsmithsBot
 text/..
wikbot/1.50 CFNetwork/548.0.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
Peachy MediaWiki Bot API Version 1.0
 application/vnd.php.serialized
 -
AnomieBOT 1.0 (TagDater)
 application/json
 text/..
MediaWiki::Bot/3.2.6
 application/json
 text/..
DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 image/..
DNSTallyKwBot/0.2
 text/..
Tawbot (public svn release; plwiki)
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
 application/json
 application/x-www-form-urlencoded
python-wikitools/1.2 (User:BernsteinBot)
 application/json
 application/x-www-form-urlencoded
 text/..
DotNetWikiBot/2.97 (Unix 2.6.32.36; )
 text/..
 -
 application/xml
AnomieBOT 1.0 (ReplaceExternalLinks2)
 application/json
 text/..
FAST Enterprise Crawler 6 used by ESP ( mail address )
 text/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/1.7.7.93) Opera Mini/3.1
 image/..
 -
 text/..
gsa-crawler (Enterprise; T3-MRHJJGX73YWBJ; mail address )
 text/..
 -
Test Webbot
 text/..
MLBot (www.metadatalabs.com/mlbot)
 text/..
 application/vnd.php.serialized
 -
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
 text/..
 -
 application/opensearchdescription+xml
SiteSeekerCrawler/1.0
 text/..
plantspedia data crawler
 text/..
COIBot/1.00
 text/..
VeeloBot 1.0
 text/..
 -
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
 image/..
 text/..
SineBot/1.5.18(User:SineBot)
 application/vnd.php.serialized
 text/..
 -
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
 text/..
 -
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
 text/..
 -
DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
SemrushBot/0.91
 text/..
 image/..
UCMore Crawler App
 text/..
 -
DotNetWikiBot/2.97 (Unix 2.6.32.37; )
 text/..
HTMLParser/1.6
 text/..
 image/..
AniBot/0.9 php/curl
 application/vnd.php.serialized
 text/..
MireoBot
 text/..
 -
 application/xml
MyCuteBot/0.1
 text/..
 application/json
 application/vnd.php.serialized
DotNetWikiBot/2.97 (Unix 5.10.0.0; )
 application/xml
 text/..
HAZY.SPIDER/Nutch-1.4
 text/..
 application/ogg
 application/pdf
JavaCrawler/1.1
 text/..
KWSS Crawler Ver. 0.1
 text/..
GoogleBot/2.1
 text/..
 image/..
 application/json
Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
 text/..
GoogleBot 2.1
 text/..
 -
SchoolReviewNetworkWikiBot
 application/json
AnomieBOT 1.0 (FlagIconRemover)
 application/json
HRoestBot, de-wikipedia using pywikipedia framework
 application/json
 application/xml
 text/..
Spinuf Spider
 text/..
 -
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
 text/..
Webwiki Search Engine Bot - www.webwiki.de
 text/..
DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
 image/..
Wiktionary spider. mail address
 text/..
DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
SiocWikiBot/1.0
 application/vnd.php.serialized
 text/..
Wikibot
 text/..
 image/..
 application/opensearchdescription+xml
OrlodrimBot/1.0
 text/..
DotNetWikiBot/2.7 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 image/..
BritannicaProjBot mail address
 text/..
Twitterbot/1.0
 text/..
 image/..
 -
 application/ogg
DotNetWikiBot/2.96 (Unix 5.10.0.0; )
 text/..
 application/xml
LinksCrawler 0.1beta
 text/..
 -
AnomieBOT 1.0 (TemplateSubster)
 application/json
HTMLParser/2.0
 text/..
Empedia Bot
 text/..
CaBot Script (running on nightshade.toolserver.org)
 application/vnd.php.serialized
 text/..
Xaldon WebSpider 2.7.b8
 text/..
XLinkBot/1.00
 text/..
FAST Enterprise Crawler 6 used by LexisNexis ( mail address )
 text/..
 -
 image/..
wikbotlite/1.50 CFNetwork/548.0.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
 text/..
 -
bitlybot
 text/..
 image/..
 -
GNAA-bot
 text/..
TrueKnowledgeBot bot mail address >
 application/vnd.php.serialized
 application/xml
microbot
 text/..
daytrippy.com Crawler
 application/json
 text/..
SurakWare MediaWiki Bot/1.0
 text/..
AnomieBOT 1.0 (PERTableUpdater)
 application/json
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer)
 application/json
DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
CorenSearchBot/1.5 en libwww-perl/6.02
 text/..
FAST Enterprise Crawler/5.3.4 ( mail address )
 text/..
 -
SemrushBot/Nutch-1.5-SNAPSHOT
 text/..
 image/..
COIBot/2.0
 text/..
CheMoBot/1.00
 text/..
MyWikipediaBot/1.0
 application/vnd.php.serialized
QCRI-Crawler/Nutch-1.4
 text/..
wikbot/1.31 CFNetwork/548.0.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
TVersity Media Robot
 text/..
TheKeens bot
 text/..
AnomieBOT 1.0 (BAGBot)
 application/json
 text/..
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
 image/..
 text/..
DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
Mozilla/5.0 (compatible; GoogleBot/2.1;
 text/..
 -
 image/..
DotNetWikiBot/2.9 (Unix 5.10.0.0; )
 text/..
Freebase Deathbot
 text/..
Goalkeeperbot(User:Beetstra)/1.0
 text/..
super cool bot
 application/vnd.php.serialized
python-wikitools/1.2 (User:LaraBot)
 application/json
wikbot/1.50 CFNetwork/485.13.9 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
 text/..
 application/xml
Alex Blokha bot/2.9 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
My Bot
 image/..
 text/..
AnomieBOT 1.0 (RandomPagePicker)
 application/json
HBC Archive Indexerbot 0.9a
 text/..
ShipCrawler/1.0
 text/..
GoogleBot
 text/..
 image/..
K-D Bot
 text/..
 -
 image/..
WikiBot/0.1
 text/..
Mozilla/5.0 (Bgbot 0.5)
 text/..
qwebiz mail address
 text/..
Mozilla 5.0 (Apibot 0.30b5)
 application/vnd.php.serialized
DotNetWikiBot/2.98 (Unix 3.0.0.12; )
 text/..
 application/xml
OrangeCrawler/Nutch-1.0 ( mail address )
 text/..
AnomieBOT 1.0 (AFDMergeFromCleaner)
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
AnomieBOT 1.0 (DeletionSortingCleaner)
 application/json
123peoplebot/1.0
 text/..
python-wikitools/1.2 (User:Mr.Z-bot)
 application/json
EarwigBot/0.1.dev (Python/2.7.1; https://github.com/earwig/earwigbot; mail address )
 application/json
IssueCrawler
 text/..
Zing-BottaBot/1.0
 text/..
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
 application/xml
R6_CommentReader(www.radian6.com/crawler)
 text/..
 -
Hexabot V1.3 - curl - api.php
 text/..
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 370 subscribers; feed-id=3852576738117026533)
 application/xml
 -
MyBachelorWorkBot/0.1
 text/..
rogerbot/1.0
 text/..
Geni ircpybot 1.0
 text/..
 application/json
 application/xml
creasybot
 application/json
15355.68total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Aug 6, 2012 15:01
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers