See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 312,023,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 2,032,561,000 external requests, which is 15.4%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	application/x-external-editor	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxy6000)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-8)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-7)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-5)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-1)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-9)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-6)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) (via babelfish.yahoo.com)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-3)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-4)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-2)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-0)
	google
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~cloudcrawling)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: my-api)
		code.google.com/appengine	text/..	mail address AppEngine-Google; (url; appid: s~wiki-sherpa)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nowwhocan)
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		code.google.com/appengine	application/x-external-editor	AppEngine-Google; (url; appid: s~cloudcrawling)
		code.google.com/appengine	-	AppEngine-Google; (url; appid: wikipedia-raw)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 6.1; zh-CN; rv:1.9.2.2) AppEngine-Google; (url; appid: fwall-w15)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: thetechnolust)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.5 KHTML Chrome/19.0.1084.56 Safari/536.5 AppEngine-Google; (url; appid: s~brightle33)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: demostene)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: your-zone)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikidashboard)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: finchproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: bubba-ps)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: bel3afya)
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ssdprox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: epvweb2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dabubad)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: calymirror)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: batamsearch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: raja584sekhar)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~clon-games)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: gateway2web)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: cravibruce)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0 AppEngine-Google; (url; appid: s~bcnof10-hrd)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ikaryse)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: gravurexgravure)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~kushgenius)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~iappsdk)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: no-restrict)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kutchix)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~mygoagenta)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nocensuraitaliana)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0 AppEngine-Google; (url; appid: donut-1)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~sherryipv6proxy)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.10 KHTML Chrome/23.0.1262.0 Safari/537.10 AppEngine-Google; (url; appid: s~dreamww1)
		code.google.com/apis/kml	text/..	Kml-Google; (url), gzip
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~xihuanlvse)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: yuricamara)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kaveriselvaraj)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: hoptheborder)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: taterproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: browsepast)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~guyu2711)
		code.google.com/appengine	text/..	Mozilla/5.0 (X11; Linux i686) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: s~suipaidoor)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nallanikrithika)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.16 KHTML Chrome/24.0.1297.0 Safari/537.16 AppEngine-Google; (url; appid: s~itunnels)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.18 KHTML Chrome/18.0.1011.0 Safari/535.18 AppEngine-Google; (url; appid: s~manmandetanqin)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~kevland3002)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~zchm123)
		code.google.com/appengine	text/..	Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:13.0) Gecko/20100101 Firefox/13.0 AppEngine-Google; (url; appid: s~zhangljaproxy)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~celinejyy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: gaucho-labnol)
		code.google.com/appengine	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~daqieqie)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: goodersearch)
		code.google.com/appengine	text/..	Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~zhzhkt06)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~newslide4you)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1; rv:10.0.10) Gecko/20100101 Firefox/10.0.10 AppEngine-Google; (url; appid: s~kmhpagent)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.41 Safari/536.5 AppEngine-Google; (url; appid: s~zero-dr)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: bypass-filter)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 KHTML Chrome/15.0.874.121 Safari/535.2 AppEngine-Google; (url; appid: edupry)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~link123451)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~wenyonghuang)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0 AppEngine-Google; (url; appid: s~freemagician002)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: guidesites)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: godfatherabhi)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 KHTML Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0 AppEngine-Google; (url; appid: s~zsqjoe5)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~baozuotun416)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikipedia-raw)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~chunxiaoyoung)
		code.google.com/appengine	text/..	Mozilla/5.0 (X11; Linux i686; rv:10.0.10) Gecko/20100101 Firefox/10.0.10 Iceweasel/10.0.10 AppEngine-Google; (url; appid: s~yxwhagent)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nicoflysurf)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~gc-nju-001)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~liningradio3)
		code.google.com/appengine	-	AppEngine-Google; (url; appid: my-api)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1 AppEngine-Google; (url; appid: s~cnbitsbackup)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: jmcim01)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0 AppEngine-Google; (url; appid: s~proxy-lvmax-xj)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~frp529)
		code.google.com/appengine	-	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nashimlive-nashimnx)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~lcrproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: calyphrox)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.12 KHTML Maxthon/3.0 Chrome/18.0.966.0 Safari/535.12 AppEngine-Google; (url; appid: s~zcl19870803)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: jackieonthefloor)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0 AppEngine-Google; (url; appid: s~josephzm1989)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~aurora-prox)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0 AppEngine-Google; (url; appid: s~aefgmno)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ax4413)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 KHTML Chrome/15.0.874.106 Safari/535.2 AppEngine-Google; (url; appid: s~phoenisake119)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.8 KHTML Chrome/23.0.1251.2 Safari/537.8 AppEngine-Google; (url; appid: s~yqwcrawl)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.14) Gecko/2009082707 Firefox/3.0.14 AppEngine-Google; (url; appid: s~flyingmsn)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0 AppEngine-Google; (url; appid: s~naplean2012)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: s~weiyouandy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: twitterchitthajagat)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~projection11111)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ridemyhell)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.22 KHTML Chrome/19.0.1049.3 Safari/535.22 AppEngine-Google; (url; appid: s~cusptea)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0 AppEngine-Google; (url; appid: s~1949zhongguo)
		code.google.com/appengine	text/..	Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~uicpt3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~hfmapplication)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 KHTML Chrome/23.0.1271.64 Safari/537.11 AppEngine-Google; (url; appid: s~huaxia283611)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: privatproxy)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.11 KHTML Chrome/20.0.1132.47 Safari/536.11 AppEngine-Google; (url; appid: s~apoloo365)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AppEngine-Google; (url; appid: s~guiguiabcd)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0 AppEngine-Google; (url; appid: s~ashen61)
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy0)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: response-out)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: images-jpg)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyfile8)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxypython7)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: tn7sub)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: p8roxy)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: cgiproxy6)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: front-pages)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: ddbrite)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: gif-images)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: down-wj)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wow-proxy)
	discoveryengine
		discoveryengine.com/discoverybot.html	text/..	Mozilla/5.0 (compatible; discoverybot/2.0; url)
		discoveryengine.com/discoverybot.html	text/..	Mozilla/5.0 (compatible; discoverybot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: image-proxy2)
		discoveryengine.com/discoverybot.html	text/..	Mozilla/5.0 (compatible; discoverybot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: cgiproxy6)
		discoveryengine.com/discoverybot.html	text/..	Mozilla/5.0 (compatible; discoverybot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxypython0)
		discoveryengine.com/discoverybot.html	text/..	Mozilla/5.0 (compatible; discoverybot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: bytearrayr)
		discoveryengine.com/discoverybot.html	text/..	Mozilla/5.0 (compatible; discoverybot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy5)
		discoveryengine.com/discoverybot.html	application/x-external-editor	Mozilla/5.0 (compatible; discoverybot/2.0; url)
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) ASProxy/5.5b4
		www.google.com/bot.html	application/x-external-editor	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Nokia2700c-2/2.0 (09.97) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (compatible; GoogleBot/2.1; url)/UCWEB8.0.3.99/90/352 UNTRUSTED/1.0
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	-	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
	yioop
		www.yioop.com/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot; url)
		www.yioop.com/bot.php	application/x-external-editor	Mozilla/5.0 (compatible; YioopBot; url)
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
		ahrefs.com/robot/	application/x-external-editor	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	-	Mozilla/5.0 (compatible; JikeSpider; url)
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b4
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
		help.naver.com/robots/	application/x-external-editor	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url)
	customernet
		quaba.customernet.de	text/..	quaba-spider (url)
	msn
		search.msn.com/msnbot.htm	text/..	msnbot/0.01 (url)
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)
	coccoc
		help.coccoc.vn/	text/..	coccoc/1.0 (url)
		help.coccoc.vn/	text/..	coccoc/1.0 (url) AppEngine-Google; (http://code.google.com/appengine; appid: surf710)
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url)
	facebook
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url)
		www.archive.org/details/archive.org_bot	-	Mozilla/5.0 (compatible; archive.org_bot url)
	genieo
		www.genieo.com/webfilter.html	text/..	Mozilla/5.0 (compatible; Genieo/1.0 url)
	wikimedia
		meta.wikimedia.org/wiki/User:Tietew	text/..	Cheebot/0.5.7 (url)
	yacy
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.6.4-1-ARCH; java 1.7.0_09; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_33; UTC/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.6.6-1-ARCH; java 1.7.0_03; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/de) url
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.0 url
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.18.0 url
		en.wikipedia.org/wiki/Wikipedia:Huggle	-	Huggle/2.1.19.0 url
	localhost
		localhost	text/..	Mozilla/5.0 (compatible; heritrix/2.0.2 url)
		localhost/reggae	text/..	WordPress/3.4.2; url
	subshell
		www.subshell.com	text/..	Mozilla/5.0 (compatible; Sophora Linkchecker; url)
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexNews/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexNewslinks; url)
	n-grams
		www.n-grams.org/icorpusbot.html	text/..	iCorpusBot (url)
	duckduckgo
		duckduckgo.com/duckduckbot.html	application/xml	DuckDuckBot/1.1; (url)
	tiscali
		www.tiscali.it/	text/..	Mozilla/5.0 (compatible; IstellaBot/1.10.2 url)
	mobileproxy
		mobileproxy.mobi	text/..	Mozilla/5.0 (compatible; MobileSurf; url)
	phonifier
		www.phonifier.com	text/..	Mozilla/5.0 (compatible; Phonifier; url)
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
	nb
		www.nb.no/vevfangst	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
	myspace
		www.myspace.com	text/..	Mozilla/5.0(compatible;MSIE/6.0url)
	pixray
		www.pixray.com/pixraybot	text/..	Pixray-Seeker/2.0 (Pixray-Seeker; url; mail address )
	creativecommons
		wiki.creativecommons.org/Metadata_Scraper	text/..	CC Metadata Scaper url
	jetsli
		jetsli.de/crawler	text/..	Mozilla/5.0 (compatible; Jetslide; url)
	www.
		www.	text/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
	bibalex
		archive.bibalex.org/bot/	text/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
	sblog
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
	vbseo
		www.vbseo.com	text/..	Mozilla/4.0 (vBSEO; url)
	mobilla
		mobilla.mobi/	text/..	MWTMC (Mobilla1.0; Website Transcoder for Mobile Clients; url)
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url)
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
	rcdtokyo
		www.rcdtokyo.com/pc2m/	text/..	Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
	parsijoo
		www.parsijoo.ir	text/..	Mozilla/5.0 (compatible; mail address url)
	w3
		www.w3.org/2006/07/mobileok-ddc	text/..	W3C-mobileOK/DDC-1.0 (see url)
	babelserver
		babelserver.org/rix	text/..	RixBot (url)
	harvard
		hul.harvard.edu/ois/digpres/projects.html	text/..	Mozilla/5.0 (compatible; special_archiver_js/3.1.2 url)
		hul.harvard.edu/ois/digpres/projects.html	text/..	Mozilla/5.0 (compatible; special_archiver/3.1.2 url)
	210
		87.151.70.210	text/..	Mozilla/5.0 (compatible; heritrix/3.1.1 url)
	matuschek
		www.matuschek.net/jobo.html	text/..	mail address (url)
		www.matuschek.net/jobo.html	text/..	JoBo/1.x (url)
	avantbrowser
		www.avantbrowser.com	text/..	Avant Browser (url)
		www.avantbrowser.com	text/..	Advanced Browser (url)
	jetbrains
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 1.0.x (url)
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 2.0 Release Candidate 1 (url)
	sistrix
		crawler.sistrix.net/	text/..	Mozilla/5.0 (compatible; SISTRIX Crawler; url)
	ostermiller
		ostermiller.org/tulipchain/	text/..	TulipChain/5.x (url) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17
		ostermiller.org/tulipchain/	text/..	TulipChain/5.xx (url) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
	expert-html
		www.expert-html.com	text/..	The Expert HTML Source Viewer (url)
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
	kula
		kula.jp/endo	text/..	endo/1.0 (Mac OS X; ppc i386; url)
	fairshare
		fairshare.cc	text/..	Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
		fairshare.cc	text/..	Mozilla crawl/5.0 (compatible; fairshare.cc url)
	career-x
		www.career-x.de/bot.html	text/..	Mozilla/5.0 (compatible; CareerBot/1.1; url)
	spidersoft
		www.spidersoft.com	text/..	WebZIP/x.x (url)
	sbl
		sbl.net	text/..	SBL-BOT (url)
	orcabrowser
		www.orcabrowser.com	text/..	Orca Browser (url)
	zipcommander
		www.zipcommander.com/	text/..	1st ZipCommander (Net) - url
	drupal
		drupal.org/	text/..	User-Agent: Drupal (url)
		drupal.org/	text/..	Drupal (url)
	sf
		liferea.sf.net/	text/..	Liferea/0.x.x (Linux; en_US.UTF-8; url)
	loc
		www.loc.gov/webarchiving/notice_to_webmasters.html	text/..	Mozilla/5.0 (compatible; special_archiver/1.5.0 url)
	zootycoon
		www.zootycoon.com	text/..	Zoo Tycoon 2 Client -- url
	ponderer
		ponderer.org/download/annotate_google.user.js	text/..	annotate_google; url
	up
		www.up.ru	text/..	Mozilla/5.0 (compatible; Mozilla/5.0 url)
	globalspec
		www.globalspec.com/Ocelli	text/..	Ocelli/1.4 (url)
	jobs
		www.jobs.de/robot.html	text/..	Mozilla/5.0 (compatible; jobs.de-Robot url)
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url)
	php
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17
		pear.php.net/	text/..	PEAR HTTP_Request class ( url )
	ibis
		ibis.ne.jp/browser/about.html	text/..	Mozilla/4.0 (compatible; ibisBrowser; url)
	grapeshot
		www.grapeshot.co.uk/crawler.php	text/..	Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url)
	sujonhera
		sujonhera.com	text/..	WordPress/3.4.2; url
	feedparser
		feedparser.org/	text/..	UniversalFeedParser/5.0.1 url
	jabse
		www.jabse.com/bot.php	text/..	Jabse.com/2.0 (url)
	tweetmeme
		tweetmeme.com/	text/..	Mozilla/5.0 (compatible; TweetmemeBot/3.0; url)
	scanmine
		www.scanmine.com	text/..	Scanmine newsspider. See url for more information. NB: We need css files!
	Anonymouse
		Anonymouse.org/	text/..	url (Unix)
	illinois
		www.clinecenter.illinois.edu/	text/..	Mozilla/5.0 (compatible; heritrix/3.1.1 url)
	instapaper
		www.instapaper.com/	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
245,902	total

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
		text/..
	Tawbot (public svn release; plwiki)
		text/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; )
		text/..
		application/xml
	DotNetWikiBot/2.100 (Unix 5.10.0.0; )
		text/..
		application/xml
	XLinkBot/1.00
		text/..
	DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		application/xml
	DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	www.integromedb.org/Crawler
		text/..
		application/x-external-editor
	DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Mozilla/5.0 (Bgbot 0.5)
		text/..
	php wikibot classes
		application/vnd.php.serialized
		-
	Bobot
		text/..
	Mozilla/5.0 (compatible; UnisterBot; mail address )
		text/..
	COIBot/1.00
		text/..
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
		text/..
		application/xml
	SurakWare MediaWiki Bot/1.0
		text/..
	DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	AsgardBot - DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	FAST Search Web Crawler 14.0.0325.0000
		text/..
	qwebiz mail address
		text/..
		application/x-external-editor
	Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlBot)
		text/..
	SineBot/1.5.19(User:SineBot)
		text/..
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
		text/..
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	IssueCrawler
		text/..
	DotNetWikiBot/2.101 (Unix 3.0.0.12; )
		text/..
	DotNetWikiBot/2.97 (Unix 5.10.0.0; )
		text/..
	DotNetWikiBot/2.101 (Unix 3.2.0.32; )
		text/..
	DotNetWikiBot/2.100 (Unix 2.6.32.38; )
		text/..
	Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
		text/..
		application/x-external-editor
	Mozilla/5.0 (compatible; EqraTechBot/1.0; mail address )
		text/..
		-
	CopperBot/0.2 [[w:de:User:P.Copp]] ( mail address )
		application/json
	DotNetWikiBot/2.100 (Unix 3.0.0.12; )
		text/..
		application/xml
	WPBot 1.0
		text/..
	NoyaBot
		text/..
	Mozilla/5.0 (Windows; Crawler; Windows NT 6.0; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7
		text/..
	Xaldon WebSpider
		text/..
	HTTPcurl.class v0.1.0-0 - Bot Sandbox cleaner v1.0 using WikiBot.class v0.1.0-0
		application/vnd.php.serialized
	JOC Web Spider
		text/..
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
		text/..
	UCMore Crawler App
		text/..
	DotNetWikiBot/2.92 (Microsoft Windows NT 6.1.7600.0; )
		text/..
	PythonWikipediaBot/1.0
		text/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	NL-Crawler
		text/..
	WordChampBot
		text/..
	DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	gsa-crawler (Enterprise; T3-P9JWVCTT9WWGY; mail address )
		text/..
	Luxobot/1.1 (toolserver; php) mail address
		text/..
	'citeseerxbot'
		text/..
	TerraSpider
		text/..
	Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
		text/..
	HosiryuhosiBot Vote Checker Report
		text/..
	spider
		text/..
	HosiryuhosiBot Vote Checker
		text/..
	DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; )
		text/..
	GoogleBot-Image/1.0
		text/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	R6_FeedFetcher(www.radian6.com/crawler)
		text/..
	Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; mail address )
		text/..
	DotNetWikiBot/2.92 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
		text/..
	DotNetWikiBot/2.92 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	DotNetWikiBot/2.100 (Unix 3.5.0.18; )
		text/..
	OrlodrimBot/1.0
		text/..
	Mozilla/5.0 (compatible; GoogleBot/2.1;
		text/..
	DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
	DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
		text/..
	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
		text/..
	Mozilla/5.0 (compatible; Mightycrawler/1.0)
		text/..
	AdMedia bot
		text/..
	AkeronBot PHP/Curl
		text/..
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
		text/..
	DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
	DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7600.0; )
		text/..
	spider-on-the-fly 20q v4.256 Discover Life www.discoverlife.org
		text/..
	eDintorni crawler
		text/..
	Goalkeeperbot(User:Beetstra)/1.0
		text/..
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		text/..
175,179	total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Sat, Nov 17, 2012 5:42
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers