See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 62,960,100 page requests (mime type text/html only!) per day are considered crawler requests, out of 476,252,070 external requests, which is 13.2%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	google
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortografia4)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: s~redconceptual)
		www.google.com/feedfetcher.html	-	FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/xml	FeedFetcher-Google; (url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien4)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: rarplayer)
		www.google.com/feedfetcher.html	application/json	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortopedianew)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien3)
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url)
		code.google.com/p/crawler4j/	text/..	crawler4j (url)
		www.google.com/coop/cse/cref	text/..	FeedFetcher-Google-CoOp; (url)
		www.google.com/feedfetcher.html	application/xml	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw)
		desktop.google.com/	text/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web-phpproxy)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: s~senchaiosrc)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	application/json	Mozilla 3.5 AppEngine-Google; (url; appid: prfleme)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~expinia-wiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~tpbitalia)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: tinysrc)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	application/json	MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~harunakaze)
		code.google.com/appengine	text/..	www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; documents; url)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: boxapp)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; url)
		www.google.com/bot.html	image/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wagagate)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: prfleme)
		code.google.com/appengine	image/..	Mozilla/5.0 (Windows; Windows NT 6.1; zh-CN; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16 ( .NET4.0E) QQDownload/1.7 AppEngine-Google; (url; appid: donut-1)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: my-reg)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: davidgotmoney50)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: freeoursouls)
		code.google.com/p/ronda	application/json	Ronda - url
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: openeyeproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pazvantoff)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kbworld24)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nelzomamirror)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tdmplong)
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: finchproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: cmd-proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: toom16-10)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: abdulfat)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: thakurproxy)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; drawings; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: threewiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tunisistan)
		code.google.com/appengine	text/..	mail address AppEngine-Google; (url; appid: itravelapp)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: demowaiy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: findadvise)
	facebook
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
		developers.facebook.com	image/..	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.0 (url)
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.0; url)
		www.google.com/bot.html	text/..	Mozilla/5.0(compatible;GoogleBot/2.1;url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	application/xml	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	application/vnd.php.serialized	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: gif-images)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: tcpudp10)
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRW/1.0 crawler (url)
		listing.yahoo.co.jp/support/faq/int/other/other_001.html	text/..	Y!J-BRJ/YATS crawler (url)
		developer.yahoo.com/yql/provider	text/..	Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	image/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.com/help/us/ysearch/slurp	application/vnd.php.serialized	Mozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRI/0.0.1 crawler ( url )
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRT/1.0 crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com	text/..	Mozilla/5.0 (YahooYSMcm/3.0.0; url)
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider-image(url)
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0(compatible;Baiduspider/2.0;url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider(url)
		www.baidu.com/search/spider.html	image/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/customer_webtxt_02.jsp	text/..	Mozilla/4.0 (compatible; NaverBot/1.0; url)
	msn
		search.msn.com/msnbot.htm	text/..	msnbot-Products/1.0 (url)
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)._
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-NewsBlogs/2.0b (url)
		search.msn.com/msnbot.htm	image/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-UDiscovery/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)._
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexDirect/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
		yandex.com/bots	application/vnd.php.serialized	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexImages/3.0; url)
	www.
		www.	text/..	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
		www.	text/..	Google - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
	sblog
		fulltext.sblog.cz/screenshot/	image/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	-	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	application/javascript	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.0; url)
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/1.0; url)
	php
		pear.php.net/	application/vnd.php.serialized	PEAR HTTP_Request class ( url )
		pear.php.net/	application/xml	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/0.5.2 (url) PHP/5.2.17
		pear.php.net/	text/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.0.0RC1 (url) PHP/5.3.2-1ubuntu4.9
		pear.php.net/	image/..	PEAR HTTP_Request class ( url )
	wordpress
		02varvara.wordpress.com	text/..	WordPress/MU; url
		iwansuwandy.wordpress.com	text/..	WordPress/MU; url
		einflussreicheleute.wordpress.com	text/..	WordPress/MU; url
		gunnyg.wordpress.com	text/..	WordPress/MU; url
		worldwright.wordpress.com	text/..	WordPress/MU; url
		josefboberg.wordpress.com	text/..	WordPress/MU; url
		godheadpost.wordpress.com	text/..	WordPress/MU; url
		alfonsopinel.wordpress.com	text/..	WordPress/MU; url
		driwancybermuseum.wordpress.com	text/..	WordPress/MU; url
		ageszagen.wordpress.com	text/..	WordPress/MU; url
		curtisnarimatsu.wordpress.com	text/..	WordPress/MU; url
		thaiintelligentnews.wordpress.com	text/..	WordPress/MU; url
		vetrinadipreghiera.wordpress.com	text/..	WordPress/MU; url
		loveandfearless.wordpress.com	text/..	WordPress/MU; url
		superockers.wordpress.com	text/..	WordPress/MU; url
		elmoderador.wordpress.com	text/..	WordPress/MU; url
		greatriversofhope.wordpress.com	text/..	WordPress/MU; url
		eof737.wordpress.com	text/..	WordPress/MU; url
		kterrl.wordpress.com	text/..	WordPress/MU; url
	traslated
		mymemory.traslated.net/doc/	text/..	Mozilla/5.0 (MyMemory Bot url)
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		toolbar.youdao.com/	image/..	Youdao Toolbar (url)
		www.youdao.com/help/webmaster/spider/	image/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		www.youdao.com/help/webmaster/spider/	application/vnd.php.serialized	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
	wwwgogetpapers
		wwwgogetpapers.com/	application/json	User-Agent: GoGetPapersBot (url)
		wwwgogetpapers.com/	text/..	User-Agent: GoGetPapersBot (url)
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.18.0 url
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WikiCleaner (url)
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.1.0 url
		en.wikipedia.org	text/..	url
		fr.wikipedia.org/wiki/Utilisateur:Salebot	application/json	Salebot, see url (uses Perl MediaWiki::API)
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle2/2.1.18 url
	sentymetr
		sentymetr.pl/bot.html	application/json	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
		sentymetr.pl/bot.html	text/..	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
	entireweb
		www.entireweb.com/about/search_tech/speedy_spider/	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
		www.entireweb.com/about/search_tech/speedy_spider/	-	Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	application/vnd.php.serialized	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	image/..	Sogou Pic Spider/3.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
	yacy
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_23; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_17; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-13-generic; java 1.6.0_23; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.38-12-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 2.6.18-028stab091.2; java 1.6.0_20; Etc/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Asia/ja) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 2.6.33.7-server-2mnb; java 1.6.0_22; Europe/fr) url
		yacy.net/bot.html	-	yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.35-30-generic; java 1.6.0_20; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet/any; amd64 Linux 2.6.38-12-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.18-274.7.1.el5; java 1.6.0_20; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/de) url
	soso
		help.soso.com/webspider.htm	text/..	Sosospider(url)
		help.soso.com/webspider.htm	-	Sosospider(url)
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url)
		www.archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; archive.org_bot url)
	covario
		www.covario.com/ids	text/..	CovarioIDS/1.1 (url; mail address )
	toolserver
		wiki.toolserver.org/view/GeoHack	text/..	Geohack (url)
		toolserver.org/~bayo/	text/..	LudoThecaire/1.0 (url)
		toolserver.org/~dispenser/	text/..	DispensersTools (url)
		toolserver.org/~guandalug/	application/vnd.php.serialized	Guandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
		toolserver.org/~para/cgi-bin/kmlexport	text/..	url libwww-perl/6.02
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url)
	wikimedia
		tools.wikimedia.de/~daniel/	text/..	WikiSense (url)
		meta.wikimedia.org/wiki/User:Tietew	text/..	Cheebot/0.5.7 (url)
	sf
		liferea.sf.net/	text/..	Liferea/0.x.x (Linux; en_US.UTF-8; url)
		magpierss.sf.net	text/..	MagpieRSS/0.7x (url)
		liferea.sf.net/	text/..	Liferea/1.x.x (Linux; es_ES.UTF-8; url)
	justsystems
		www.justsystems.com/jp/tech/crawler/	text/..	JUST-CRAWLER(url)
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	image/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	text/..	jikespider (Mozilla/5.0 (compatible; JikeSpider; url))
		shoulu.jike.com/spider.html	text/..	jikespider (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	text/..	jikespider ( (compatible; JikeSpider; url))
	tumblr
		benderthewebrobot.tumblr.com	text/..	Mozilla/5.0 (compatible; Bender; url)
		benderthewebrobot.tumblr.com	application/vnd.php.serialized	Mozilla/5.0 (compatible; Bender; url)
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url)
		help.zum.com/inquiry	image/..	ZumBot/1.0 (ZUM Search; url)
	enwp
		enwp.org/User:SDPatrolBot	text/..	SDPatrolBot (url)
		enwp.org/User:KingpinBot	text/..	KingpinBot (url)
		enwp.org/User:H3llkn0wz/WikiSharpAPI	text/..	WikiSharpAPI/0.3 url (C# .NET)
	jetbrains
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 2.0 Release Candidate 1 (url)
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 1.0.x (url)
	avantbrowser
		www.avantbrowser.com	text/..	Advanced Browser (url)
		www.avantbrowser.com	text/..	Avant Browser (url)
	mediawiki
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url)
	feedshow
		www.feedshow.com	text/..	FeedshowOnline (url)
		www.feedshow.com	text/..	Feedshow/x.0 (url; 1 subscriber)
	newsgator
		www.newsgator.com/	text/..	FeedDemon/2.7 (url; Microsoft Windows XP)
		www.newsgator.com	text/..	NewsGatorOnline/2.0 (url; 1 subscribers)
	bin-co
		www.bin-co.com/php/scripts/load/	text/..	BinGet/1.00.A (url)
		www.bin-co.com/php/scripts/load/	application/vnd.php.serialized	BinGet/1.00.A (url)
		www.bin-co.com/php/scripts/load/	application/xml	BinGet/1.00.A (url)
	wikidict
		www.wikidict.de	text/..	url
	z-add
		w3.z-add.co.uk/linkcheck/	text/..	Z-Add Link Checker (url)
	federatedmedia
		federatedmedia.net	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	kosmix
		www.kosmix.com/html/kosmos.html	application/xml	Mozilla/5.0(compatible;Kosmos/1.0;url)
	garlik
		garlik.com/	text/..	GarlikCrawler/1.1 (url, mail address )
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url)
	echonest
		the.echonest.com/reader/	application/xml	nestReader/0.3 (discovery; url; reader at echonest.com)
		the.echonest.com/reader/	text/..	nestReader/0.3 (discovery; url; reader at echonest.com)
	bsurprised
		bsurprised.com/	text/..	BSurprised WikiBox 0.1.3 (url)
	flipboard
		flipboard.com/browserproxy	image/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	application/json	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
	daum
		ws.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
		ws.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
	emining
		emining.jp/	text/..	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
		emining.jp/	-	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
	semager
		www.semager.de/blog/semager-bots/	text/..	Mozilla/5.0 (compatible; Semager/1.4; url)
	tinyurl
		tinyurl.com/64t5n	text/..	Rome Client (url) Ver: 0.9
		tinyurl.com/64t5n	application/xml	Rome Client (url) Ver: UNKNOWN
	seebot
		seebot.org	text/..	Lynx/2.8 (;url)
	bne
		www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html	text/..	Mozilla/5.0 (compatible; archive.org_bot/1.5.0 url)
		www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html	image/..	Mozilla/5.0 (compatible; archive.org_bot/1.5.0 url)
	tweetmeme
		tweetmeme.com/	text/..	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
	rssreader
		www.rssreader.com	text/..	RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
	ranchero
		ranchero.com/netnewswire/	text/..	NetNewsWire/2.x (Mac OS X; url)
	ponderer
		ponderer.org/download/annotate_google.user.js	text/..	annotate_google; url
	plagger
		plagger.org/	text/..	Plagger/0.x.xx (url)
	kula
		kula.jp/endo	text/..	endo/1.0 (Mac OS X; ppc i386; url)
	zootycoon
		www.zootycoon.com	text/..	Zoo Tycoon 2 Client -- url
	graemef
		graemef.com	text/..	NewsGator FetchLinks extension/0.2.0 (url)
	rssbandit
		www.rssbandit.org	text/..	RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
	winpodder
		winpodder.com	text/..	WinPodder (url)
	timewe
		timewe.net	text/..	CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
	orcabrowser
		www.orcabrowser.com	text/..	Orca Browser (url)
	nemui
		mozshot.nemui.org/	text/..	Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
	it-influentials
		search.it-influentials.com/bot.htm	text/..	Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
	snarfware
		www.snarfware.com/	text/..	Snarfer/0.x.x (url)
	zipcommander
		www.zipcommander.com/	text/..	1st ZipCommander (Net) - url
	blogbridge
		www.blogbridge.com/	text/..	BlogBridge 2.13 (url)
	freebase
		www.freebase.com	text/..	metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
	feeds4all
		www.feeds4all.com/feedzcollector	text/..	FeedZcollector v1.x (Platinum) url
	sistrix
		crawler.sistrix.net/	text/..	Mozilla/5.0 (compatible; SISTRIX Crawler; url)
	github
		github.com/pauldix/typhoeus/tree/master	text/..	Typhoeus - url
		github.com/NeilCrosby/wikislurp	application/vnd.php.serialized	WikiSlurp (url)
	bibalex
		archive.bibalex.org/bot/	image/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
		archive.bibalex.org/bot/	text/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
	whatrhymeswith
		www.whatrhymeswith.com/site/rhyme-bot	text/..	RhymeBot/0.1 (url)
	4chat
		www.4chat.tv	text/..	url
	SearchNearMe
		SearchNearMe.com/contact.php	application/vnd.php.serialized	SearchNearMe (url)
		SearchNearMe.com/contact.php	text/..	SearchNearMe (url)
	hatena
		a.hatena.ne.jp/help	text/..	Hatena Antenna/0.5 (url)
	apercite
		www.apercite.fr/robot/index.html	image/..	Mozilla/5.0 (compatible; Apercite; url)
	Anonymouse
		Anonymouse.org/	text/..	url (Unix)
		Anonymouse.org/	image/..	url (Unix)
		Anonymouse.org/	application/ogg	url (Unix)
	ac
		www.cse.iitb.ac.in/~vishaal_h4	text/..	DrRajendra/Nutch-0.9 (IIT Kharagpur; url; mail address )
		www.yazduni.ac.ir	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		www.cse.iitb.ac.in/~vishaal_h4	text/..	Amit/Nutch-0.9 (IIT Kharagpur; url; mail address )
		www.clips.ua.ac.be/pages/pattern	text/..	Pattern/1.0 url
	whstour
		tokyo.whstour.com	text/..	WordPress/3.2.1; url
		osaka.whstour.com	text/..	WordPress/3.2.1; url
		nagoya.whstour.com	text/..	WordPress/3.2.1; url
	speaktoit
		www.speaktoit.com	application/json	Speaktoit url
	fairshare
		fairshare.cc	text/..	Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
		fairshare.cc	text/..	Mozilla crawl/5.0 (compatible; fairshare.cc url)
		fairshare.cc	application/vnd.php.serialized	Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
	goo
		help.goo.ne.jp/contact/	text/..	goo wikipedia (url)
		help.goo.ne.jp/help/article/1142/	-	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
		help.goo.ne.jp/help/article/1142/	text/..	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
	netnewswireapp
		netnewswireapp.com/mac/	-	NetNewsWire/3.3 (Mac OS X; url; gzip-happy)
	rediff
		pages.rediff.com	text/..	Rediff Pages (url)
		pages.rediff.com	image/..	Rediff Pages (url)
	alexa
		www.alexa.com/site/help/webmasters	text/..	ia_archiver (url; mail address )
	weblio
		www.weblio.jp/	text/..	Mozilla/5.0 (compatible; WeblioBot; url)
	simplepie
		simplepie.org	application/xml	SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
		simplepie.org	text/..	SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
	wikiglass
		wikiglass.com	text/..	url : mail address
	apache
		lucene.apache.org/nutch/bot.html	text/..	NutchCVS/0.7.2 (Nutch; url; mail address )
	spinn3r
		spinn3r.com/robot	text/..	Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
	drupal
		drupal.org/	text/..	User-Agent: Drupal (url)
		drupal.org/	text/..	Drupal (url)
		drupal.org/	text/..	Drupal (url
	moviecus
		www.moviecus.com/botcontactinfo.php	application/yaml	moviecus bot (url)
	yioop
		www.yioop.com/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot url)
		yioop.com/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot url)
	textdigger
		textdigger.com	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	metamagazine
		metamagazine.com	text/..	WordPress/3.2.1; url
	acordocoletivo
		acordocoletivo.org	text/..	WordPress/MU; url
	rockpeaks
		www.rockpeaks.com/contact	text/..	RockPeaks/0.1 (url)
	puritysearch
		www.puritysearch.net/	text/..	Mozilla/5.0 (compatible; Purebot/1.1; url)
	suggy
		blog.suggy.com/was-ist-suggy/suggy-webcrawler/	text/..	Mozilla/5.0 (compatible; suggybot v0.01a, url)
		blog.suggy.com/was-ist-suggy/suggy-webcrawler/	-	Mozilla/5.0 (compatible; suggybot v0.01a, url)
	netvibes
		www.netvibes.com	text/..	Netvibes (url)
	mytvmoments
		www.mytvmoments.com	text/..	My TV Moments (url)
	pannous
		pannous.info	text/..	Mozilla/5.0 (Voice Actions url)
		pannous.net	text/..	Mozilla/5.0 (Voice Actions url)
	vik
		vik.com	text/..	vik-robot/Nutch-1.0 (vikspider; url; mail address )
	froute
		labs.froute.jp/pc2m/help.html	text/..	Froute Mobile Gateway/1.0 (url)
	searchtechnologies
		www.searchtechnologies.com	text/..	Mozilla/5.0 (compatible; heritrix/1.14.3 url)
	ibis
		ibis.ne.jp/browser/about.html	image/..	Mozilla/4.0 (compatible; ibisBrowser; url)
		ibis.ne.jp/browser/about.html	text/..	Mozilla/4.0 (compatible; ibisBrowser; url)
	blogscope
		www.blogscope.net/	text/..	Mozilla/5.0 (compatible; BlogScope/1.0; url; U of Toronto)
	embed
		support.embed.ly/	image/..	Mozilla/5.0 (compatible; Embedly/0.2; url)
		support.embed.ly/	text/..	Mozilla/5.0 (compatible; Embedly/0.2; url)
	trendiction
		www.trendiction.de/bot	text/..	Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
	annauniv
		www.annauniv.edu	text/..	AUCEG/Nutch-0.9 (url; mail address )
	sourceforge
		linkchecker.sourceforge.net/	text/..	LinkChecker/7.2 (url)
	seokicks
		www.seokicks.de/robot.html	text/..	Mozilla/5.0 (compatible; SEOkicks-Robot url)
	arquivo
		arquivo.pt/faq-crawling	text/..	Arquivo-web-crawler (compatible; heritrix/1.14.3 url)
	plagiarismcheck
		plagiarismcheck.org	application/json	WikiCrawl 1.0b (url contact-mail: mail address )
	archive-it
		archive-it.org/files/site-owners.html	image/..	Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
		archive-it.org/files/site-owners.html	text/..	Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
		archive-it.org/files/site-owners.html	-	Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
	js-kit
		js-kit.com/	text/..	JS-Kit URL Resolver, url
	paper
		support.paper.li/entries/20023257-what-is-paper-li	text/..	Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
	zapbot
		www.zapbot.com	text/..	Mozilla/5.0 (compatible; ZapBot/0.2c; url)
		www.zapbot.net	text/..	Mozilla/5.0 (compatible; ZapBot/0.2n; url)
		www.zapbot.org	text/..	Mozilla/5.0 (compatible; ZapBot/0.2o; url)
	scoutjet
		www.scoutjet.com/	text/..	Mozilla/5.0 (compatible; ScoutJet; url)
	bazqux
		crawler.bazqux.com	text/..	BazQux Crawler (url; mail address )
	abonti
		www.abonti.com	text/..	Mozilla/5.0 (compatible; Abonti/0.91 - url)
	search
		www.search.ch/rim.html	text/..	UltraSpider3000/1.0 (url)
	creativecommons
		wiki.creativecommons.org/Metadata_Scraper	text/..	CC Metadata Scaper url
		wiki.creativecommons.org/Metadata_Scraper	text/..	CC Metadata Scaper url
	mytake
		dt1.mytake.jp/bot.html	text/..	mytakebot/0.9 (url)
	instapaper
		www.instapaper.com/	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
	topsy
		labs.topsy.com/butterfly/	text/..	Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
88564.6599999976	total

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	PythonWikipediaBot/1.0
		application/json
		application/xml
		text/..
		image/..
		-
	GoogleBot-Image/1.0
		text/..
		image/..
		-
	MediaWikiCrawler-Google/2.0 ( mail address )
		text/..
		-
	ClueBot/1.1
		application/vnd.php.serialized
		-
	php wikibot classes
		application/vnd.php.serialized
		text/..
		-
	Answersbot
		text/..
		-
	LinkParser/2.0
		text/..
	GoogleBot-Image/1.0
		text/..
		image/..
		application/vnd.php.serialized
		-
		application/json
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
		text/..
		-
		application/vnd.php.serialized
		application/ogg
	wikiwix-bot-3.0
		text/..
		-
	WikiBookBot/0.1
		text/..
	Peachy MediaWiki Bot API Version 1.0
		application/vnd.php.serialized
		-
		text/..
	Onespot Crawler
		application/json
		text/..
		-
	ClueBot/2.0
		application/vnd.php.serialized
		text/..
	spider
		text/..
		image/..
		application/json
	Pywikipediabot/2.0
		application/json
		text/..
	AarghBot Linux
		text/..
	MoovidaBot/0.1
		text/..
		-
	mail address
		application/vnd.php.serialized
		text/..
	NameSpider/1.0
		text/..
		image/..
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
		text/..
		application/vnd.php.serialized
	Mozilla 5.0 (Apibot 0.32)
		application/vnd.php.serialized
		text/..
	wikiBot Ver0.1
		application/json
	jikespider "Mozilla/5.0
		text/..
		image/..
		-
		application/xml
		application/ogg
	DigitalsmithsBot
		text/..
	YBot/0.1
		application/vnd.php.serialized
		text/..
	DotNetWikiBot/2.97 (Unix 5.10.0.0; )
		text/..
		application/xml
	MediaWiki::Bot/3.2.6
		application/json
		text/..
	AnomieBOT 1.0 (TagDater)
		application/json
	python-wikitools/1.2 (User:BernsteinBot)
		application/json
		text/..
	GoogleBot
		text/..
		-
		image/..
	Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
		text/..
		-
		image/..
		application/opensearchdescription+xml
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		image/..
		text/..
		application/json
		-
	Test Webbot
		text/..
		-
	strucr.com crawler 0.7.69 (refer to in robots.txt as strucr, see https://strucr.com/bot)
		text/..
	Mozilla/4.0 (compatible; EmberSpider 0.8; Scout (a); bgft)
		text/..
	Metabot 0.1
		text/..
	UCMore Crawler App
		text/..
		-
	AnomieBOT 1.0 (ReplaceExternalLinks2)
		application/json
		text/..
	Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
		text/..
		-
	Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
		text/..
		-
	SemrushBot/0.9
		text/..
	HTMLParser/2.0
		text/..
	Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
		text/..
		application/rsd+xml
		-
		application/opensearchdescription+xml
	t_crawler/0.4
		text/..
		image/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.5.1.0) Opera Mini/3.1
		image/..
		text/..
	upictoBot
		text/..
		image/..
	FAST Enterprise Crawler 6 used by ESP ( mail address )
		text/..
	AniBot/0.9 php/curl
		application/vnd.php.serialized
		-
	MLBot (www.metadatalabs.com/mlbot)
		text/..
		application/vnd.php.serialized
	Webwiki Search Engine Bot - www.webwiki.de
		text/..
	DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		image/..
	jikespider "
		image/..
		text/..
	SpinSpider
		text/..
	Code Search Crawler/Nutch-1.2 (Code Search Crawler; www.iai.uni-bonn.de)
		text/..
	Mozilla/4.0 /Nutch-1.0 (robot_nutch_ics_ict; mail address )
		text/..
		-
		application/ogg
		image/..
	HRoestBot, de-wikipedia using pywikipedia framework
		application/json
		application/xml
		text/..
	SineBot/1.5.18(User:SineBot)
		application/vnd.php.serialized
		text/..
	MyCuteBot/0.1
		text/..
		application/json
		application/vnd.php.serialized
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
		text/..
	TheKeens bot
		text/..
	COIBot/2.0
		text/..
	Tawbot (public svn release; plwiki)
		text/..
	strucr.com crawler 0.6.62 (refer to in robots.txt as strucr, see https://strucr.com/bot)
		text/..
	yolinkBot
		text/..
	Twitterbot/0.1
		text/..
		image/..
	AnomieBOT 1.0 (FlagIconRemover)
		application/json
	strucr.com crawler 0.7.68 (refer to in robots.txt as strucr, see https://strucr.com/bot)
		text/..
	useragent: WikiBot/0.1
		text/..
		application/vnd.php.serialized
	AdMedia bot
		text/..
		-
	SiocWikiBot/1.0
		application/vnd.php.serialized
		text/..
	CaBot Script (running on nightshade.toolserver.org)
		application/vnd.php.serialized
		text/..
	DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
		text/..
	TVersity Media Robot
		text/..
	COIBot/1.00
		text/..
	FAST Search Web Crawler 14.0.0291.0000
		text/..
	plantspedia data crawler
		text/..
	SurakWare MediaWiki Bot/1.0
		text/..
		application/xml
	Mozilla/5.0 (compatible; SelazBot/4.2)
		text/..
		-
	wikbot/1.23 CFNetwork/548.0.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
		application/json
		text/..
	MediaWiki::Bot/v3.4.2
		application/json
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
		application/x-www-form-urlencoded
	AnomieBOT 1.0 (TemplateSubster)
		application/json
	GNAA-bot
		text/..
	DotNetWikiBot/2.96 (Unix 5.10.0.0; )
		text/..
		application/xml
	Twitterbot/1.0
		text/..
		image/..
	SineBot/1.5.17(User:SineBot)
		application/vnd.php.serialized
		text/..
	wikbot/1.23 CFNetwork/548.0.3 Darwin/11.0.0
		image/..
		application/json
		-
		text/..
	FAST Enterprise Crawler 6 used by viaapia (viaapia)
		text/..
		-
	AnomieBOT 1.0 (OrphanReferenceFixer)
		application/json
	TrueKnowledgeBot bot mail address >
		application/vnd.php.serialized
		application/xml
		text/..
	DNSTallyKwBot/0.2
		text/..
	jikespider ("Mozilla/5.0)
		text/..
		-
		application/ogg
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
		image/..
		text/..
	AnomieBOT 1.0 (BAGBot)
		application/json
		text/..
	SchoolReviewNetworkWikiBot
		application/json
		text/..
	DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		-
	XLinkBot/1.00
		text/..
	SearchBot
		text/..
		application/xml
		application/vnd.php.serialized
	IssueCrawler
		text/..
	Hopperbot-Image/1.0
		image/..
		text/..
	CheMoBot/1.00
		text/..
	Spinuf Spider
		text/..
	Wiktionary spider. mail address
		text/..
	Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
		application/xml
		text/..
	Slevnicka.cz CURL bot
		text/..
	OrlodrimBot/1.0
		text/..
	Wikibot 1.50 (Macintosh; Mac OS X 10.7.2; de_AT)
		image/..
		text/..
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.5.1.0) Opera Mini/3.1
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.5.0.0) Opera Mini/3.1
		image/..
		text/..
	Handelabra WikiBot
		application/vnd.php.serialized
		text/..
	HTMLParser/1.4
		text/..
	Peachy MediaWiki Bot API Version 0.1beta
		application/vnd.php.serialized
	Baiduspider
		text/..
	unblockbot/1.00
		text/..
	lssbot
		text/..
		application/xml
	OrangeCrawler/Nutch-1.0 ( mail address )
		text/..
	HTMLParser/1.6
		text/..
	Wikibot
		text/..
		image/..
		-
	Mozilla/5.0 QunarBot/1.0
		text/..
	Freebase Deathbot
		text/..
	bitlybot
		text/..
		image/..
	DotNetWikiBot/2.9 (Unix 5.10.0.0; )
		text/..
	DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 371 subscribers; feed-id=3852576738117026533)
		application/xml
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.6.0.0) Opera Mini/3.1
		image/..
		text/..
	My Bot
		image/..
		text/..
		-
		application/ogg
	wikbot/1.23 CFNetwork/485.13.9 Darwin/11.0.0
		image/..
		application/json
		text/..
	DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
		text/..
	SearQuBot/SearQuBot v1.0
		text/..
		application/ogg
	python-wikitools/1.2 (User:LaraBot)
		application/json
	NFCCheckBot/1.0
		text/..
	BotMapDev/1.3.677 CFNetwork/548.0.3 Darwin/11.0.0
		image/..
	HBC Archive Indexerbot 0.9a
		text/..
	Crawler/0.0 (Crawler using Nutch 1.3; ur EMAIL Addr here)
		text/..
	BotMapDev/1.3.683 CFNetwork/548.0.3 Darwin/11.0.0
		image/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.4.0.0) Opera Mini/3.1
		image/..
		text/..
	Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7)
		text/..
	My Bot
		text/..
	FAST Enterprise Crawler 6 used by LexisNexis ( mail address )
		text/..
	PadosAttilaCrawler/Nutch-1.0 (Ozi,PolandWiz,AustriaWiz,WiennaWiz crawlers, Attila Pados, mail address ; www.ozi.hu, www.polandwiz.com,www.wiennawiz.com,www.austriawiz.com; attila dot mail address )
		text/..
	AnomieBOT 1.0 (AFDMergeFromCleaner)
		application/json
	AnomieBOT 1.0 (RandomPagePicker)
		application/json
	Mozilla 5.0 (Apibot 0.30b5)
		application/vnd.php.serialized
	Geni ircpybot 1.0
		text/..
		application/json
		application/xml
	BrittainBot/1.0
		text/..
	microbot
		text/..
	kmSearchBot
		text/..
	Jabse.com Crawler v.2.0 www.jabse.com/crawler.php
		text/..
	Mozilla/5.0 (Bgbot 0.5)
		text/..
	strucr.com crawler 0.7.64 (refer to in robots.txt as strucr, see https://strucr.com/bot)
		text/..
18908.91	total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Aug 6, 2012 13:10
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers