See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 98,936,520 page requests (mime type text/html only!) per day are considered crawler requests, out of 545,156,840 external requests, which is 18.1%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	google
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/feedfetcher.html	-	FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/xml	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortografia4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortopedianew)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: rarplayer)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~cloudcrawling)
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien3)
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		code.google.com/p/crawler4j/	text/..	crawler4j (url)
		www.google.com/feedfetcher.html	application/json	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		desktop.google.com/	text/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: s~redconceptual)
		www.google.com/feedfetcher.html	application/xml	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien4)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen)
		code.google.com/appengine	text/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
		code.google.com/appengine	-	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; documents; url)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~kasumiremix)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki2)
		code.google.com/appengine	text/..	Wiki.java 0.27 AppEngine-Google; (url; appid: wikipediatools)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki4)
		www.google.com/bot.html	application/ogg	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		www.google.com/coop/cse/cref	text/..	FeedFetcher-Google-CoOp; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~hr-pulsesubscriber)
		www.google.com/bot.html	NONE/wikipedia	- Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		code.google.com/appengine	text/..	Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: boxapp)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~app3123ak)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: usawebproxy0)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: goodersearch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: worldwide-propaganda)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~theunblock)
		www.google.com/feedfetcher.html	-	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~wikigraph2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~drizzlprox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~francetiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebproxy0)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kires-roxy)
		code.google.com/appengine	image/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki8)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: threewiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: my-api)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pakgalaxy)
		code.google.com/appengine	text/..	www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~misterhac)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki6)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dkoxyserv)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki7)
		code.google.com/appengine	application/json	MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki1)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: 114proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: guidesites)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki5)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: yourbudgets)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: boxapp)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: toom16-10)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: thetitoowebproxy)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	text/..	mail address AppEngine-Google; (url; appid: s~wiki-sherpa)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: atxproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vi-mobile)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~proxyseekkety)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: davrasaurs)
		code.google.com/appengine	application/json	Mozilla/5.0 AppEngine-Google; (url; appid: s~redconceptual)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: mehproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxyusing121)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: your-zone)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: thetechnolust)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ivegotalovelybunch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tusawebproxy4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wmhsonline)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxy-devakishor)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nagarajhubli-proxy-server)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web4proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pazvantoff)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: adrianswebproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: python-proxy-server)
	facebook
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.0 (url)
		developers.facebook.com	image/..	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	application/json	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.0 (url)
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	application/vnd.php.serialized	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	application/json	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	application/ogg	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
		www.bing.com/bingbot.htm	application/xml	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk)
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	application/json	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	GoogleBot/2.1 (url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	application/ogg	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	application/json	Mozilla/5.0 (compatible; YandexBot/3.0; url)
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRW/1.0 crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	image/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRJ/YATS crawler (url)
		developer.yahoo.com/yql/provider	text/..	Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRT/1.0 crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	application/xml	Mozilla/5.0 (compatible; Yahoo! Slurp;url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url)
		www.baidu.com/search/spider.htm	image/..	Baiduspider-image(url)
		www.baidu.com/search/spider.html	application/json	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider-image(url)
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	text/..	Yeti/1.1 (NHN Corp.; url)
		help.naver.com/robots/	application/json	Yeti/1.0 (NHN Corp.; url)
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
		ahrefs.com/robot/	-	Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
		ahrefs.com/robot/	application/json	Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
		ahrefs.com/robot/	application/ogg	Mozilla/5.0 (compatible; AhrefsBot/4.0; url)
	msn
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	image/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-UDiscovery/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-Products/1.0 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-NewsBlogs/2.0b (url)
		search.msn.com/msnbot.htm	image/..	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	image/..	msnbot-NewsBlogs/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot/0.01 (url)
		search.msn.com/msnbot.htm	-	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	image/..	msnbot-Products/1.0 (url)
		search.msn.com/msnbot.htm	-	msnbot/2.0b (url)
	cibra
		cibra.de/	text/..	CiBra Data Collector (url)
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
	genieo
		www.genieo.com/webfilter.html	text/..	Mozilla/5.0 (compatible; Genieo/1.0 url)
		www.genieo.com/webfilter.html	application/xml	Mozilla/5.0 (compatible; Genieo/1.0 url)
		www.genieo.com/webfilter.html	image/..	Mozilla/5.0 (compatible; Genieo/1.0 url)
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		toolbar.youdao.com/	image/..	Youdao Toolbar (url)
	finecomb
		finecomb.com/	application/json	api/1.1 (url; mail address )
		finecomb.com/	-	api/1.1 (url; mail address )
	sblog
		fulltext.sblog.cz/screenshot/	image/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	-	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	-	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/screenshot/	application/ogg	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url)
		help.zum.com/inquiry	image/..	ZumBot/1.0 (ZUM Search; url)
	php
		pear.php.net/	application/vnd.php.serialized	PEAR HTTP_Request class ( url )
		pear.php.net/	text/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/0.5.2 (url) PHP/5.2.17
		pear.php.net/	image/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	application/xml	HTTP_Request2/2.0.0 (url) PHP/5.3.8
		pear.php.net/	application/xml	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17
		pear.php.net/package/http_request2	image/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15
	soso
		help.soso.com/webspider.htm	text/..	Mozilla/5.0(compatible; Sosospider/2.0; url)
		help.soso.com/webspider.htm	text/..	Sosospider(url)
		help.soso.com/webspider.htm	-	Sosospider(url)
		help.soso.com/webspider.htm	-	Mozilla/5.0(compatible; Sosospider/2.0; url)
	wordpress
		fotosdeatrizesemodelos.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
		cgagneux.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
		josefboberg.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
		greatriversofhope.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
		klausgauger.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
		tsjok45.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
		lesliebrodie.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
		jochenlembke.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
		fotosdeatrizesemodelos.wordpress.com	text/..	WordPress/3.5-RC5-23155; url
		barbielistholland.wordpress.com	text/..	WordPress/3.5-RC6-23166; url
	www.
		www.	text/..	GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
		www.	image/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
	echonest
		the.echonest.com/reader/	application/xml	nestReader/0.3 (discovery; url; reader at echonest.com)
		the.echonest.com/reader/	text/..	nestReader/0.3 (discovery; url; reader at echonest.com)
	coccoc
		help.coccoc.vn/	text/..	coccoc/1.0 (url)
		help.coccoc.vn/	-	coccoc/1.0 (url)
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url)
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
		www.exabot.com/go/robot	-	Mozilla/5.0 (compatible; Exabot/3.0; url)
	blekko
		blekko.com/about/blekkobot	text/..	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
		blekko.com/about/blekkobot	-	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
		blekko.com/about/blekkobot	image/..	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
	143
		173.13.143.74/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot; url)
		173.13.143.74/bot.php	image/..	Mozilla/5.0 (compatible; YioopBot; url)
		173.13.143.74/bot.php	-	Mozilla/5.0 (compatible; YioopBot; url)
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.0 url
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WPCleaner (url)
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19 url
	yacy
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-45-server; java 1.6.0_26; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.5.7-gentoo; java 1.6.0_24; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-308.8.2.el5.028stab101.1; java 1.6.0_26; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.4.9-gentoo; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.5.0-19-generic; java 1.7.0_09; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (Superarama-Beta/any; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.5.0-21-generic; java 1.7.0_09; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.1.10-1.16-default; java 1.6.0_24; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.4.11-2.16-desktop; java 1.7.0_09; Europe/nl) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.6.4-1-ARCH; java 1.7.0_09; Europe/fr) url
		yacy.net/bot.html	-	yacybot (freeworld/global; i386 Linux 3.5.0-21-generic; java 1.7.0_09; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_04; America/en) url
		yacy.net/bot.html	text/..	yacybot (webportal/global; i386 Linux 3.2.0-33-generic; java 1.6.0_24; America/en) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 3.6.4-1-ARCH; java 1.7.0_09; Europe/fr) url
		yacy.net/bot.html	-	yacybot (webportal/global; i386 Linux 3.2.0-33-generic; java 1.6.0_24; America/en) url
		yacy.net/bot.html	-	yacybot (freeworld/global; i386 Linux 3.5.0-19-generic; java 1.7.0_09; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; x86_64 Mac OS X 10.8.2; java 1.6.0_37; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; x86_64 Mac OS X 10.8.2; java 1.6.0_37; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-34-generic; java 1.6.0_24; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld-global; x86 Windows 7 6.1; java 1.7.0_05; America/pt) url
		yacy.net/bot.html	-	yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 2.6.32-042stab061.2; java 1.7.0_09; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/en) url
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	application/json	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou inst spider/4.0(url)
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	-	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	image/..	Mozilla/5.0 (compatible; JikeSpider; url)
	wikidict
		www.wikidict.de	text/..	url
	toolserver
		wiki.toolserver.org/view/GeoHack	text/..	Geohack (url)
		toolserver.org/~dispenser/	image/..	CacheThumbs/1.2 (url)
		toolserver.org/~dispenser/	text/..	DispensersTools (url)
		toolserver.org/~dispenser/	text/..	CacheThumbs/1.2 (url)
		toolserver.org/~para/cgi-bin/kmlexport	text/..	url libwww-perl/6.02
		toolserver.org/~dispenser/	application/json	DispensersTools (url)
	traslated
		mymemory.traslated.net/doc/	text/..	Mozilla/5.0 (MyMemory Bot url)
	flipboard
		flipboard.com/browserproxy	image/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	application/json	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
		flipboard.com/browserproxy	image/..	null (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	-	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	-	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url)
		www.archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; archive.org_bot url)
		archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url)
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url)
	bin-co
		www.bin-co.com/php/scripts/load/	text/..	BinGet/1.00.A (url)
		www.bin-co.com/php/scripts/load/	application/vnd.php.serialized	BinGet/1.00.A (url)
	goo
		help.goo.ne.jp/contact/	text/..	goo wikipedia (url)
		help.goo.ne.jp/door/crawler.html	text/..	ichiro/3.0 (url)
		goo.gl/7y4SX	text/..	GoogleProducer; (url)
		search.goo.ne.jp/option/use/sub4/sub4-1/	text/..	ichiro/3.0 (url)
		search.goo.ne.jp/option/use/sub4/sub4-1/	-	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
	yioop
		www.yioop.com/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot; url)
		www.yioop.com/bot.php	image/..	Mozilla/5.0 (compatible; YioopBot; url)
	okian
		www.okian.ro/	text/..	MyBot/1.0 (url)
	wwwgogetpapers
		wwwgogetpapers.com/	application/json	User-Agent: GoGetPapersBot (url)
	ephorus
		www.ephorus.com/	text/..	Mozilla/5.0 (compatible; Ephorusbot/1.4.5.6; url)
	enwp
		enwp.org/User:SDPatrolBot	text/..	SDPatrolBot (url)
		enwp.org/User:KingpinBot	text/..	KingpinBot (url)
		enwp.org/User:H3llkn0wz/WikiSharpAPI	text/..	WikiSharpAPI/0.3 url (C# .NET)
	onet
	SearchNearMe
		SearchNearMe.com/contact.php	application/vnd.php.serialized	SearchNearMe (url)
		SearchNearMe.com/contact.php	text/..	SearchNearMe (url)
	dataparksearch
		dataparksearch.org/bot	text/..	DataparkSearch/4.54-26052011 (url)
	daum
		tab.search.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
	cognarius
		cognarius.com	application/json	AppsArlak/1.0 (url)
		cognarius.com	text/..	AppsArlak/1.0 (url)
	zeebox
		www.zeebox.com	text/..	Zeebox (url)
		www.zeebox.com	application/json	Zeebox (url)
	proximic
		www.proximic.com/info/spider.php	text/..	Mozilla/5.0 (compatible; proximic; url)
	topsy
		labs.topsy.com/butterfly/	text/..	Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
	kosmix
		www.kosmix.com/html/kosmos.html	application/xml	Mozilla/5.0(compatible;Kosmos/1.0;url)
	sistrix
		crawler.sistrix.net/	text/..	Mozilla/5.0 (compatible; SISTRIX Crawler; url)
	wita
		www.wita.de	text/..	WITA/nutchbot/Nutch-1.5 (url; mail address )
	wikiglass
		wikiglass.com	text/..	url : mail address
	ac
		www.tkl.iis.u-tokyo.ac.jp/~crawler/	text/..	Mozilla/5.0 (compatible; Steeler/3.5; url)
		www.ninjal.ac.jp/corpus_center/ulc/crawl-en	text/..	Mozilla/5.0 (compatible; heritrix/3.1.1 url)
	toshiba
		www.toshiba.co.jp/rdc/about/crawl_info.htm	text/..	TosCrawler/Nutch-1.4 (url; ' mail address dot co dot jp')
		www.toshiba.co.jp/rdc/about/crawl_info_en.htm	text/..	TosCrawler/Nutch-1.4 (url; ' mail address dot co dot jp')
		www.toshiba.co.jp/rdc/about/crawl_info.htm	text/..	TosCrawler/Nutch-1.6 (url; ' mail address dot co dot jp')
		www.toshiba.co.jp/rdc/about/crawl_info.htm	text/..	TosCrawler/Nutch-1.5.1 (url; ' mail address dot co dot jp')
	xbmc
		www.xbmc.org	image/..	XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
		www.xbmc.org	text/..	XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
	speaktoit
		www.speaktoit.com	application/json	Speaktoit url
	apercite
		www.apercite.fr/robot/index.html	image/..	Mozilla/5.0 (compatible; Apercite; url)
	drupal
		drupal.org/	image/..	Drupal (url)
		drupal.org/	text/..	Drupal (url)
		drupal.org/	text/..	User-Agent: Drupal (url)
	gamedipper
		www.gamedipper.com	application/json	gamedipper.com bot (url)
	federatedmedia
		federatedmedia.net	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	bibalex
		archive.bibalex.org/bot/	text/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
		archive.bibalex.org/bot/	image/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
	paper
		support.paper.li/entries/20023257-what-is-paper-li	text/..	Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
	zipcode
		zipcode.us	text/..	Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
	emining
		emining.jp/	text/..	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
		emining.jp/	-	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
	wikimpress
		wikimpress.org/	text/..	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
		wikimpress.org/	-	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
	tineye
		tineye.com/crawler.html	application/json	TinEye/1.1 (url)
	seokicks
		www.seokicks.de/robot.html	text/..	Mozilla/5.0 (compatible; SEOkicks-Robot url)
	weblio
		www.weblio.jp/info/crawler.jsp	image/..	Mozilla/5.0 (compatible; Webliobot/0.1; url)
		www.weblio.jp/	text/..	Mozilla/5.0 (compatible; WeblioBot; url)
		www.weblio.jp/info/crawler.jsp	text/..	Mozilla/5.0 (compatible; Webliobot/0.1; url)
	picsearch
		www.picsearch.com/bot.html	text/..	psbot/0.1 (url)
		www.picsearch.com/bot.html	image/..	psbot/0.1 (url)
	github
		github.com/pauldix/typhoeus/tree/master	text/..	Typhoeus - url
		github.com/pauldix/feedzirra/tree/master	application/xml	feedzirra url
		github.com/edsu/linkypedia	application/json	linkpyediabot v0.1: url
	rcdtokyo
		www.rcdtokyo.com/pc2m/	text/..	Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
		www.rcdtokyo.com/pc2m/	-	Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
	embed
		support.embed.ly/	image/..	Mozilla/5.0 (compatible; Embedly/0.2; snap; url)
		support.embed.ly/	text/..	Mozilla/5.0 (compatible; Embedly/0.2; url)
	hatena
		a.hatena.ne.jp/help	text/..	Hatena Antenna/0.5 (url)
	localhost:8888
		localhost:8888	image/..	WordPress/3.5; url
		localhost:8888	text/..	WordPress/3.5; url
	freecoon
		www.freecoon.com/	text/..	FreecoonBot/1.0 (url)
	mail
		go.mail.ru/help/robots	text/..	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url)
		go.mail.ru/help/robots	image/..	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url)
	easybib
		content.easybib.com/autocite/	text/..	EasyBib AutoCite (url)
		content.easybib.com/autocite/	application/json	EasyBib AutoCite (url)
	stackoverflow
		stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-php	text/..	Testing for url
	discoveryengine
		discoveryengine.com/discoverybot.html	text/..	Mozilla/5.0 (compatible; discoverybot/2.0; url)
	yoursite
		yoursite.com/botinfo	text/..	Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
	netarkivet
		netarkivet.dk/webcrawler/	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		netarkivet.dk/webcrawler/	image/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
	openindex
		www.openindex.io/en/webmasters/spider.html	text/..	Mozilla/5.0 (compatible; OpenindexSpider; url)
	bsurprised
		bsurprised.com/	text/..	BSurprised WikiBox 0.1.3 (url)
	sf
		liferea.sf.net/	text/..	Liferea/0.x.x (Linux; en_US.UTF-8; url)
		liferea.sf.net/	text/..	Liferea/1.x.x (Linux; es_ES.UTF-8; url)
		magpierss.sf.net	text/..	MagpieRSS/0.7x (url)
	muso
		www.muso.com	text/..	Mozilla/5.0 (compatible; musobot/1.0; mail address ; url)
	abonti
		www.abonti.com	text/..	Mozilla/5.0 (compatible; Abonti/0.91 - url)
	netseer
		www.netseer.com/crawler.html	text/..	Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
	tweetmeme
		tweetmeme.com/	text/..	Mozilla/5.0 (compatible; TweetmemeBot/3.0; url)
	plagiarismcheck
		plagiarismcheck.org	application/json	WikiCrawl 1.0b (url contact-mail: mail address )
	alexa
		www.alexa.com/site/help/webmasters	text/..	ia_archiver (url; mail address )
	superfeedr
		superfeedr.com	application/xml	Superfeedr bot/2.0 url - Please get in touch if we are polling too hard.
		superfeedr.com	text/..	Superfeedr bot/2.0 url - Please get in touch if we are polling too hard.
		superfeedr.com	-	Superfeedr bot/2.0 url - Please get in touch if we are polling too hard.
	in
		www.m-culture.in.th	text/..	m-culture.in.th (url)
	plos
		alm.plos.org	application/json	PLoS Article Level Metrics - url
	warebay
		www.warebay.com/bot.html	text/..	Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
	netvibes
		www.netvibes.com	text/..	Netvibes (url)
	rockpeaks
		www.rockpeaks.com/contact	text/..	RockPeaks/0.1 (url)
	textdigger
		textdigger.com	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	spinn3r
		spinn3r.com/robot	text/..	Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
	elcidharth
		elcidharth.com	text/..	WordPress/3.5-RC6-23166; url
	vermagerd
		www.vermagerd.be/wp	text/..	WordPress/3.4.2; url
	wiktionary
		en.wiktionary.org/wiki/User:Rukhabot	application/json	Rukhabot/0.1 (url)
	publicknowledgeproject
		alm.publicknowledgeproject.org	application/json	Article Level Metrics - url
	igrec
		www.igrec.ca/projects	text/..	Wikitionary Text Parser 0.2 (url)
	sentymetr
		sentymetr.pl/bot.html	application/json	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
		sentymetr.pl/bot.html	text/..	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
	avantbrowser
		www.avantbrowser.com	text/..	Avant Browser (url)
		www.avantbrowser.com	text/..	Advanced Browser (url)
	fotopedia
		www.fotopedia.com	application/json	Picor (url)
	moviecus
		www.moviecus.com/botcontactinfo.php	application/yaml	moviecus bot (url)
	friendofrenia
		friendofrenia.com/	application/json	User-Agent: FriendoFrenia (url)
		friendofrenia.com/	text/..	User-Agent: FriendoFrenia (url)
	jetbrains
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 2.0 Release Candidate 1 (url)
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 1.0.x (url)
	rockmelt
		rockmelt.com	text/..	RockmeltEmbedService (url)
	newsgator
		www.newsgator.com/	text/..	FeedDemon/2.7 (url; Microsoft Windows XP)
		www.newsgator.com	text/..	NewsGatorOnline/2.0 (url; 1 subscribers)
	searchtechnologies
		www.searchtechnologies.com	text/..	Mozilla/5.0 (compatible; heritrix/1.14.3 url)
	diffbot
		www.diffbot.com	image/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url)
		www.diffbot.com	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url)
	grapeshot
		www.grapeshot.co.uk/crawler.php	text/..	Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url)
	tiscali
		www.tiscali.it/	text/..	Mozilla/5.0 (compatible; IstellaBot/1.10.2 url)
	feedshow
		www.feedshow.com	text/..	Feedshow/x.0 (url; 1 subscriber)
		www.feedshow.com	text/..	FeedshowOnline (url)
	simplepie
		simplepie.org	application/xml	SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
		simplepie.org	application/xml	SimplePie/1.2.1 (Feed Parser; url; Allow like Gecko) Build/20111015034325
	mediawiki
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url)
	esciudad
		www.esciudad.com/	application/json	Esciudad/1.0 (url)
	pingdom
		www.pingdom.com/	text/..	Pingdom.com_bot_version_1.4_(url)
		www.pingdom.com	text/..	Pingdom.com_bot_version_1.4_(url)
	nb
		www.nb.no/vevfangst	image/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		www.nb.no/vevfangst	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
	duckduckgo
		duckduckgo.com/duckduckbot.html	text/..	DuckDuckBot/1.1; (url)
		duckduckgo.com/duckduckpreview.html	text/..	DuckDuckPreview/1.0; (url)
		duckduckgo.com/duckduckpreview.html	-	DuckDuckPreview/1.0; (url)
	Anonymouse
		Anonymouse.org/	image/..	url (Unix)
		Anonymouse.org/	text/..	url (Unix)
	stad
		stad.com	text/..	Mozilla/5.0 (compatible; stadbot/1.0; url)
	turnitin
		www.turnitin.com/robot/crawlerinfo.html	text/..	TurnitinBot/2.1 (url)
	instapaper
		www.instapaper.com/	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
	kindsight
		www.kindsight.net/en/kscrawler	text/..	KSCrawler/Nutch-1.0 (url; mail address )
		www.kindsight.net/en/kscrawler	text/..	KSCrawler/Nutch-1.5.1 (url; mail address )
	thomasy
		map.thomasy.tw	application/json	Thomasy Map (url)
	astropin
		astropin.com	text/..	WordPress/3.5; url
		astropin.com	image/..	WordPress/3.5; url
	microsystools
		www.microsystools.com/products/sitemap-generator/	text/..	A1 Sitemap Generator/4.1.0 (url) miggibot
	culturadigital
	backgroundswitcher
		www.backgroundswitcher.com/	text/..	John's Background Switcher 4.6 (url)
		www.backgroundswitcher.com/	image/..	John's Background Switcher 4.4 (url)
	jetsli
		jetsli.de/crawler	text/..	Mozilla/5.0 (compatible; Jetslide; url)
	tweetedtimes
		tweetedtimes.com	text/..	Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url)
		tweetedtimes.com	text/..	TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url)
	dasdonkey
		www.dasdonkey.com	text/..	Mozilla/5.0 (compatible; DonkeyBot/0.1; url)
	js-kit
		js-kit.com/	text/..	JS-Kit URL Resolver, url
	tinyurl
		tinyurl.com/64t5n	text/..	Rome Client (url) Ver: 0.9
	sonyericsson
		www.sonyericsson.com/UAprof/R800xR301.xml	image/..	Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
	semager
		www.semager.de/blog/semager-bots/	text/..	Mozilla/5.0 (compatible; Semager/1.4c; url)
	localhost
		localhost	text/..	Mozilla/5.0 (compatible; heritrix/2.0.2 url)
	site-shot
		www.site-shot.com/	image/..	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34
126753.109999988	total

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	PythonWikipediaBot/1.0
		application/json
		application/xml
		-
		text/..
		application/x-www-form-urlencoded
		image/..
	spider
		text/..
		application/vnd.php.serialized
		application/yaml
		application/json
		-
		image/..
		application/ogg
	AniBot/0.9 php/curl
		application/vnd.php.serialized
		-
		text/..
	php wikibot classes
		application/vnd.php.serialized
		text/..
		application/json
	MediaWikiCrawler-Google/2.0 ( mail address )
		text/..
		-
	GoogleBot-Image/1.0
		image/..
		text/..
		-
	LinkParser/2.0
		text/..
		-
	Peachy MediaWiki Bot API Version 1.0
		application/vnd.php.serialized
	Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
		text/..
	wikiwix-bot-3.0
		text/..
		-
	GoogleBot-Image/1.0
		text/..
		image/..
		-
		application/json
		application/xml
		application/ogg
	gsa-crawler (Enterprise; T3-P9JWVCTT9WWGY; mail address , mail address )
		text/..
		-
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
		text/..
		-
		application/json
	tigerbot
		application/json
		text/..
	SearchBot
		text/..
	ClueBot/1.1
		application/vnd.php.serialized
	Pywikipediabot/2.0
		application/json
	DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	Answersbot
		text/..
	ClueBot/2.0
		application/vnd.php.serialized
	www.integromedb.org/Crawler
		text/..
		-
		image/..
		application/pdf
		application/xml
		application/ogg
	Wikipath Bot (email: mail address )
		application/json
	DotNetWikiBot/2.101 (Unix 2.6.32.39; )
		text/..
		application/xml
	Mozilla 5.0 (Apibot 0.32)
		application/vnd.php.serialized
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
		text/..
		application/json
		-
		image/..
	JohnFLBot/1.1
		application/vnd.php.serialized
	TrueKnowledgeBot bot mail address >
		application/xml
		application/vnd.php.serialized
	YBot/0.1
		application/vnd.php.serialized
	DigitalsmithsBot
		text/..
	Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
		text/..
		-
		application/json
		image/..
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		image/..
		text/..
		application/json
		-
	MediaWiki::Bot/3.2.6
		application/json
	CorenSearchBot/1.7 en libwww-perl/6.04
		text/..
	User-Agent: (Researcher, Bot Newbie) .NET Bot, mail address
		application/json
	AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
		application/json
	mail address mail address – MediaWiki Tcl Bot Framework 0.5
		application/json
		application/x-www-form-urlencoded
	DotNetWikiBot/2.100 (Unix 2.6.32.38; )
		text/..
	mail address
		application/vnd.php.serialized
		text/..
	DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		image/..
		application/ogg
		application/pdf
	Wikibot/2.0.1 CFNetwork/609 Darwin/13.0.0
		image/..
		application/json
		text/..
		-
	MediaWiki::Bot/3.005002
		application/json
		text/..
	Tawbot (public svn release; plwiki)
		text/..
	Web Crawler
		text/..
		-
	plantspedia data crawler
		text/..
	dtSearchSpider
		text/..
	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
		text/..
		image/..
	www.monit24.pl-m24Bot/4.0-
		-
		image/..
		text/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; )
		text/..
		application/xml
	AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
		application/json
	WikiPlaysBot
		text/..
	SineBot/1.5.19(User:SineBot)
		application/vnd.php.serialized
		text/..
	Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0
		image/..
		application/json
		text/..
	DotNetWikiBot/2.100 (Unix 5.10.0.0; )
		text/..
		application/xml
	GermCrawler
		application/json
		text/..
	SchoolReviewNetworkWikiBot
		application/json
		text/..
	AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
		application/json
	HN Spider/Nutch-2.1
		text/..
		application/ogg
	lexiapp-crawler/0.1.0 ( mail address )
		text/..
	parsijoo-crawler
		text/..
		image/..
		application/ogg
		application/xml
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	360spider-image
		image/..
		text/..
	AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
		application/json
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/x-www-form-urlencoded
		application/xml
	Phantom.js bot
		image/..
		text/..
	wikbotlite/2.0 CFNetwork/609 Darwin/13.0.0
		image/..
		application/json
		text/..
	DotNetWikiBot/2.101 (Unix 3.2.0.34; )
		text/..
	HosiryuhosiBot IRC-RecentChanges Checker
		text/..
		application/x-www-form-urlencoded
	MyCuteBot/0.1
		text/..
		application/json
	lexiwords-crawler/0.1.0 ( mail address )
		text/..
	Twitterbot/1.0
		text/..
		image/..
		-
		application/pdf
	JavaCrawler/1.1
		text/..
	SurakWare MediaWiki Bot/1.0
		text/..
		application/xml
	OrlodrimBot/1.0
		text/..
		-
		application/x-www-form-urlencoded
	AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
		application/json
		text/..
	HTMLParser/2.0
		text/..
		-
	SiocWikiBot/1.0
		application/vnd.php.serialized
		text/..
	mySpider/Nutch-1.5.1
		text/..
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
		text/..
	HRoestBot, de-wikipedia using pywikipedia framework
		text/..
		application/json
	Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl
		text/..
		-
	Bot
		text/..
	Test Webbot
		text/..
	AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
		application/json
		text/..
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
		image/..
		text/..
		application/json
	DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	TVersity Media Robot
		text/..
	Wikibot/2.0.1 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
	COIBot/1.00
		text/..
	Zing-BottaBot/2.0
		text/..
	Mozilla/5.0 (compatible; UnisterBot; mail address )
		text/..
		-
	UCMore Crawler App
		text/..
	Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
		text/..
	My Nutch Spider/Nutch-1.5
		text/..
		image/..
		application/ogg
	Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
		text/..
		-
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1)
		application/json
	XLinkBot/1.00
		text/..
	Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
		text/..
		-
	DotNetWikiBot/2.100 (Unix 3.0.0.12; )
		text/..
		application/xml
	Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	COIBot/2.0
		text/..
	HTMLParser/1.6
		text/..
		-
	GoogleBot
		text/..
		image/..
		-
	MyBot ( mail address )
		text/..
	Metabot 0.1
		text/..
	Peachy MediaWiki Bot API Version 0.1beta
		application/vnd.php.serialized
	Mozilla/5.0 (X11; Linux x86_64) Ubuntu/12.04 Codebot/1.0
		text/..
		image/..
	python-wikitools/1.2 (User:BernsteinBot)
		application/json
	WikiBot/0.1
		text/..
		image/..
	EarwigBot/0.2.dev.git4ff7612a (Python/2.7.3; https://github.com/earwig/earwigbot; mail address )
		application/json
		-
		text/..
		application/x-www-form-urlencoded
	theWxitBot/0.1
		application/json
	DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	ToyStory Crawl uk.ac.dur.ddfw58 Dissertation Crawl
		text/..
	DotNetWikiBot/2.101 (Unix 3.0.0.12; )
		text/..
		application/xml
	python-wikitools/1.2 (User:LaraBot)
		application/json
	IssueCrawler
		text/..
	WikiBot 0.1/ Email : mail address
		application/json
	Empedia Bot
		text/..
	Evolution Crawler
		text/..
		-
		image/..
		application/ogg
	AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
		application/json
	Mozilla/5.0 (Bgbot 0.5)
		text/..
	Mozilla 5.0 (Apibot 0.30b5)
		application/vnd.php.serialized
	My Nutch Spider/Nutch-1.6
		text/..
	AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]])
		application/json
	DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; )
		text/..
		application/xml
	bitlybot
		text/..
		image/..
		-
	MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
		text/..
	Mozilla/5.0 (compatible; Tbot/1.0;)
		text/..
	SINA_ROBOT; Mozilla/5.0 (Windows; Windows NT 5.1; MSIE8.0; zh-CN; rv:1.9.1.8) Gecko/20100202 Firef8
		text/..
	robert bot
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.2/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	LauschenBot/1.0 ( mail address )
		text/..
	AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]])
		application/json
	AsgardBot - DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	Anomebot v2.0
		application/json
		text/..
	bot: fr-anal
		application/json
29608.73	total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Sat, Mar 9, 2013 5:04
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers