See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 68,092,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 460,381,000 external requests, which is 14.8%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	google
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/feedfetcher.html	-	FeedFetcher-Google; (url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	text/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortografia4)
		www.google.com/feedfetcher.html	application/json	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/xml	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortopedianew)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien4)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: s~redconceptual)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien3)
		www.google.com/feedfetcher.html	-	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/xml	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: rarplayer)
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: s~senchaiosrc)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~cloudcrawling)
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw)
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/coop/cse/cref	text/..	FeedFetcher-Google-CoOp; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: abdulfat)
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
		code.google.com/appengine	-	AppEngine-Google; (url; appid: s~senchaiosrc)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: myproxywx)
		code.google.com/appengine	text/..	Wiki.java 0.25 AppEngine-Google; (url; appid: wikipediatools)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		code.google.com/appengine	application/json	Mozilla 4.0 AppEngine-Google; (url; appid: prfleme)
		code.google.com/p/crawler4j/	text/..	crawler4j (url)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	crawlr AppEngine-Google; (url; appid: s~google.com:crawlr-staging)
		code.google.com/appengine	-	AppEngine-Google; (url; appid: s~rain-soul)
		code.google.com/appengine	application/json	MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
		code.google.com/appengine	text/..	www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pakgalaxy)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~deutiki)
		code.google.com/p/ronda	application/json	Ronda - url
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; documents; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: boxapp)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlp6)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kires-roxy)
		www.google.com/bot.html	image/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~drizzlprox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: 100thpriest)
		code.google.com/appengine	-	AppEngine-Google; (url; appid: s~rain-soul2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: azamasmadi)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: misc-tools)
		code.google.com/appengine	text/..	Mozilla/5.0 AppEngine-Google; (url; appid: s~app3123)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki2)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kbworld24)
		docs.google.com	-	Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxy2031)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: prfleme)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxy-devakishor)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~proxyseekkety)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wagagate)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: openeyeproxy)
		code.google.com/appengine	-	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nation4india)
		code.google.com/appengine	image/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: cmd-proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~kyaysarlay)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: jptaravellahighschool)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kaveriselvaraj)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlp8)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: bie99miracle)
		docs.google.com	-	Mozilla/5.0 (compatible; GoogleDocs; documents; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nashimlive-nashimnx)
		code.google.com/p/crawler4j/	-	crawler4j (url)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vebproxy)
		code.google.com/appengine	-	AppEngine-Google; (url; appid: s~rain-soul4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: threewiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: worldwide-propaganda)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web4proxy)
	facebook
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
		developers.facebook.com	image/..	facebookplatform/1.0 (url)
		developers.facebook.com	-	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.1 (url)
		developers.facebook.com	text/..	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	application/vnd.php.serialized	facebookexternalhit/1.1 (url)
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	application/vnd.php.serialized	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b4
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html))
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	application/xml	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1 url)
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	-	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		listing.yahoo.co.jp/support/faq/int/other/other_001.html	text/..	Y!J-BRJ/YATS crawler (url)
		developer.yahoo.com/yql/provider	text/..	Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	application/vnd.php.serialized	Mozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	image/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRI/0.0.1 crawler ( url )
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRT/1.0 crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider-image(url)
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider(url)
		www.baidu.com/search/spider.htm	-	Baiduspider-image(url)
		www.baidu.com/search/spider.html	image/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexDirect/3.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexNewslinks; url)
		yandex.com/bots	application/vnd.php.serialized	Mozilla/5.0 (compatible; YandexBot/3.0; url)
	msn
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)._
		search.msn.com/msnbot.htm	text/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-NewsBlogs/2.0b (url)
		search.msn.com/msnbot.htm	image/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-Products/1.0 (url)
		search.msn.com/msnbot.htm	-	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-UDiscovery/2.0b (url)
		search.msn.com/msnbot.htm	-	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	-	msnbot/2.0b (url)._
	wwwgogetpapers
		wwwgogetpapers.com/	application/json	User-Agent: GoGetPapersBot (url)
		wwwgogetpapers.com/	text/..	User-Agent: GoGetPapersBot (url)
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	-	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	image/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
	sblog
		fulltext.sblog.cz/screenshot/	image/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	-	SeznamBot/3.0 (url)
	cibra
		cibra.de/	text/..	CiBra Data Collector (url)
	php
		pear.php.net/	application/vnd.php.serialized	PEAR HTTP_Request class ( url )
		pear.php.net/	application/xml	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/0.5.2 (url) PHP/5.2.17
		pear.php.net/	text/..	PEAR HTTP_Request class ( url )
		pear.php.net/	image/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	application/xml	HTTP_Request2/2.0.0 (url) PHP/5.3.8
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.14
		pear.php.net/	-	PEAR HTTP_Request class ( url )
	www.
		www.	text/..	GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	image/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		toolbar.youdao.com/	image/..	Youdao Toolbar (url)
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		www.youdao.com/help/webmaster/spider/	application/vnd.php.serialized	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
	echonest
		the.echonest.com/reader/	application/xml	nestReader/0.3 (discovery; url; reader at echonest.com)
		the.echonest.com/reader/	text/..	nestReader/0.3 (discovery; url; reader at echonest.com)
	dataparksearch
		dataparksearch.org/bot	text/..	DataparkSearch/4.54-26052011 (url)
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	application/vnd.php.serialized	Sogou web spider/4.0(url)
	soso
		help.soso.com/webspider.htm	text/..	Sosospider(url)
		help.soso.com/webspider.htm	-	Sosospider(url)
		help.soso.com/webspider.htm	application/xml	Sosospider(url)
	yacy
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.9; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_30; Asia/ja) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.42.12-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/ro) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.42.9-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
		yacy.net/bot.html	-	yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/en) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 3.2.9; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-16-server; java 1.6.0_23; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 2.6.32-5-486; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/fi) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) url
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.0 url
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WikiCleaner (url)
		en.wikipedia.org/	text/..	CSResearch/Nutch-1.2 (Research - Natural Language Engineering and Web Applications; url; mail address )
		en.wikipedia.org	text/..	url
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19 url
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.18.0 url
	entireweb
		www.entireweb.com/about/search_tech/speedy_spider/	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
	discoveryengine
		discoveryengine.com/discobot.html	text/..	Mozilla/5.0 (compatible; discobot/2.0; url)
		discoveryengine.com/discobot.html	image/..	Mozilla/5.0 (compatible; discobot/2.0; url)
		discoveryengine.com/discobot.html	-	Mozilla/5.0 (compatible; discobot/2.0; url)
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url)
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; special_archiver/3.1.1 url)
		archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
		www.archive.org/details/archive.org_bot	-	Mozilla/5.0 (compatible; archive.org_bot url)
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url)
		www.exabot.com/go/robot	-	Mozilla/5.0 (compatible; Exabot/3.0; url)
	toolserver
		wiki.toolserver.org/view/GeoHack	text/..	Geohack (url)
		toolserver.org/~dispenser/	text/..	DispensersTools (url)
		toolserver.org/~dispenser/	application/json	DispensersTools (url)
		toolserver.org/~para/cgi-bin/kmlexport	text/..	url libwww-perl/6.02
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	-	Mozilla/5.0 (compatible; JikeSpider; url)
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
		www.gnip.com/	-	UnwindFetchor/1.0 (url)
		www.gnip.com/	image/..	UnwindFetchor/1.0 (url)
	blekko
		blekko.com/about/blekkobot	text/..	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
		blekko.com/about/blekkobot	-	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
	enotes
		www.enotes.com	text/..	eNotesBot 2.0 (url)
		www.enotes.com	image/..	eNotesBot 2.0 (url)
		www.enotes.com	-	eNotesBot 2.0 (url)
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url)
		help.zum.com/inquiry	image/..	ZumBot/1.0 (ZUM Search; url)
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
	flipboard
		flipboard.com/browserproxy	image/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	-	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	application/json	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
	bin-co
		www.bin-co.com/php/scripts/load/	text/..	BinGet/1.00.A (url)
		www.bin-co.com/php/scripts/load/	application/vnd.php.serialized	BinGet/1.00.A (url)
	github
		github.com/pauldix/typhoeus/tree/master	text/..	Typhoeus - url
		github.com/NeilCrosby/wikislurp	application/vnd.php.serialized	WikiSlurp (url)
	traslated
		mymemory.traslated.net/doc/	text/..	Mozilla/5.0 (MyMemory Bot url)
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url)
	wikidict
		www.wikidict.de	text/..	url
	enwp
		enwp.org/User:SDPatrolBot	text/..	SDPatrolBot (url)
		enwp.org/User:KingpinBot	text/..	KingpinBot (url)
		enwp.org/User:H3llkn0wz/WikiSharpAPI	text/..	WikiSharpAPI/0.3 url (C# .NET)
	patriotradionetwork
		www.patriotradionetwork.com	text/..	WordPress/3.3.1; url
		www.patriotradionetwork.com	text/..	WordPress/3.3.2; url
	sf
		liferea.sf.net/	text/..	Liferea/0.x.x (Linux; en_US.UTF-8; url)
		magpierss.sf.net	text/..	MagpieRSS/0.7x (url)
		liferea.sf.net/	text/..	Liferea/1.x.x (Linux; es_ES.UTF-8; url)
	ephorus
		www.ephorus.com/	text/..	Mozilla/5.0 (compatible; Ephorusbot/1.2.2; url)
	goo
		help.goo.ne.jp/contact/	text/..	goo wikipedia (url)
		help.goo.ne.jp/door/crawler.html	text/..	ichiro/3.0 (url)
		search.goo.ne.jp/option/use/sub4/sub4-1/	text/..	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
	tweetmeme
		tweetmeme.com/	text/..	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
		tweetmeme.com/	-	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
	federatedmedia
		federatedmedia.net	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	kosmix
		www.kosmix.com/html/kosmos.html	application/xml	Mozilla/5.0(compatible;Kosmos/1.0;url)
	daum
		tab.search.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
		ws.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
	wordpress
		klima47.wordpress.com	text/..	WordPress/3.4-beta1; url
		omadeon.wordpress.com	text/..	WordPress/3.4-beta1; url
		driwancybermuseum.wordpress.com	text/..	WordPress/3.4-alpha-20343; url
	zeebox
		www.zeebox.com	application/json	Zeebox (url)
	avantbrowser
		www.avantbrowser.com	text/..	Advanced Browser (url)
		www.avantbrowser.com	text/..	Avant Browser (url)
	feedshow
		www.feedshow.com	text/..	FeedshowOnline (url)
		www.feedshow.com	text/..	Feedshow/x.0 (url; 1 subscriber)
	jetbrains
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 2.0 Release Candidate 1 (url)
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 1.0.x (url)
	wikiglass
		wikiglass.com	text/..	url : mail address
	newsgator
		www.newsgator.com/	text/..	FeedDemon/2.7 (url; Microsoft Windows XP)
		www.newsgator.com	text/..	NewsGatorOnline/2.0 (url; 1 subscribers)
	emining
		emining.jp/	text/..	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
		emining.jp/	-	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
	SearchNearMe
		SearchNearMe.com/contact.php	application/vnd.php.serialized	SearchNearMe (url)
		SearchNearMe.com/contact.php	text/..	SearchNearMe (url)
	wikimpress
		wikimpress.org/	text/..	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
	apache
		lucene.apache.org/nutch/bot.html	text/..	NutchCVS/0.7.2 (Nutch; url; mail address )
	freebase
		www.freebase.com	text/..	metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
	bibalex
		archive.bibalex.org/bot/	text/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
		archive.bibalex.org/bot/	image/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
	apercite
		www.apercite.fr/robot/index.html	image/..	Mozilla/5.0 (compatible; Apercite; url)
	speaktoit
		www.speaktoit.com	application/json	Speaktoit url
	xbmc
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (iOS; 11.0.0 AppleTV2,1, Version 5.0.1 (Build 9A406a); url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1 (Build 9B179b); url)
	whatrhymeswith
		www.whatrhymeswith.com/site/rhyme-bot	text/..	RhymeBot/0.1 (url)
	bnf
		www.bnf.fr/fr/outils/a.dl_web_capture_robot.html	text/..	Mozilla/5.0 (compatible; bnf.fr_bot; url)
		www.bnf.fr/fr/outils/a.dl_web_capture_robot.html	image/..	Mozilla/5.0 (compatible; bnf.fr_bot; url)
		www.bnf.fr/fr/outils/a.dl_web_capture_robot.html	-	Mozilla/5.0 (compatible; bnf.fr_bot; url)
	netseer
		www.netseer.com/crawler.html	text/..	Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
	kr:6600
		www.checkprivacy.or.kr:6600/RS/PRIVACY_FAQ.jsp	text/..	url
		www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp	text/..	url
	veveo
		corporate.veveo.net/webmasters.html	text/..	Mozilla/5.0 (compatible; Veveobot; url)
	archive-it
		archive-it.org/files/site-owners.html	image/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
		archive-it.org/files/site-owners.html	-	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
		archive-it.org/files/site-owners.html	text/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
	hatena
		a.hatena.ne.jp/help	text/..	Hatena Antenna/0.5 (url)
	kalooga
		kalooga.com/crawler	image/..	Mozilla/5.0 (compatible; KaloogaBot; url)
		kalooga.com/crawler	text/..	Mozilla/5.0 (compatible; KaloogaBot; url)
	bsurprised
		bsurprised.com/	text/..	BSurprised WikiBox 0.1.3 (url)
	easybib
		content.easybib.com/autocite/	text/..	EasyBib AutoCite (url)
		content.easybib.com/autocite/	application/json	EasyBib AutoCite (url)
	whstour
		whstour.com/tokyo	text/..	WordPress/3.3.1; url
		whstour.com/osaka	text/..	WordPress/3.3.1; url
		whstour.com/nagoya	text/..	WordPress/3.3.1; url
	warebay
		www.warebay.com/bot.html	text/..	Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
	trendiction
		www.trendiction.de/bot	text/..	Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
	zipcommander
		www.zipcommander.com/	text/..	1st ZipCommander (Net) - url
	abonti
		www.abonti.com	text/..	Mozilla/5.0 (compatible; Abonti/0.91 - url)
	timewe
		timewe.net	text/..	CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
	graemef
		graemef.com	text/..	NewsGator FetchLinks extension/0.2.0 (url)
	feeds4all
		www.feeds4all.com/feedzcollector	text/..	FeedZcollector v1.x (Platinum) url
	tinyurl
		tinyurl.com/64t5n	text/..	Rome Client (url) Ver: 0.9
	zootycoon
		www.zootycoon.com	text/..	Zoo Tycoon 2 Client -- url
	seebot
		seebot.org	text/..	Lynx/2.8 (;url)
	snarfware
		www.snarfware.com/	text/..	Snarfer/0.x.x (url)
	rssbandit
		www.rssbandit.org	text/..	RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
	kula
		kula.jp/endo	text/..	endo/1.0 (Mac OS X; ppc i386; url)
	alexa
		www.alexa.com/site/help/webmasters	text/..	ia_archiver (url; mail address )
	blogbridge
		www.blogbridge.com/	text/..	BlogBridge 2.13 (url)
	textdigger
		textdigger.com	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	orcabrowser
		www.orcabrowser.com	text/..	Orca Browser (url)
	it-influentials
		search.it-influentials.com/bot.htm	text/..	Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
	ranchero
		ranchero.com/netnewswire/	text/..	NetNewsWire/2.x (Mac OS X; url)
	rssreader
		www.rssreader.com	text/..	RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
	winpodder
		winpodder.com	text/..	WinPodder (url)
	metamoji
		www.metamoji.com/jp/crawler.html	text/..	Mozilla/5.0 (compatible; MetamojiCrawler/1.0; url
	nemui
		mozshot.nemui.org/	text/..	Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
	plagger
		plagger.org/	text/..	Plagger/0.x.xx (url)
	ponderer
		ponderer.org/download/annotate_google.user.js	text/..	annotate_google; url
	plagiarismcheck
		plagiarismcheck.org	application/json	WikiCrawl 1.0b (url contact-mail: mail address )
	mediawiki
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url)
	superfeedr
		superfeedr.com	application/xml	Superfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
	rockpeaks
		www.rockpeaks.com/contact	text/..	RockPeaks/0.1 (url)
	tiscali
		www.tiscali.it/	text/..	Mozilla/5.0 (compatible; IstellaBot/1.01.18 url)
	drupal
		drupal.org/	text/..	User-Agent: Drupal (url)
		drupal.org/	text/..	Drupal (url)
	duckduckgo
		duckduckgo.com/duckduckbot.html	text/..	DuckDuckBot/1.1; (url)
		duckduckgo.com/duckduckpreview.html	text/..	DuckDuckPreview/1.0; (url)
		duckduckgo.com/duckduckpreview.html	-	DuckDuckPreview/1.0; (url)
	paper
		support.paper.li/entries/20023257-what-is-paper-li	text/..	Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
	spinn3r
		spinn3r.com/robot	text/..	Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
	netnewswireapp
		netnewswireapp.com/mac/	-	NetNewsWire/3.3 (Mac OS X; url; gzip-happy)
	simplepie
		simplepie.org	application/xml	SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
		simplepie.org	text/..	SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
	searchtechnologies
		www.searchtechnologies.com	text/..	Mozilla/5.0 (compatible; heritrix/1.14.3 url)
	embed
		support.embed.ly/	text/..	Mozilla/5.0 (compatible; Embedly/0.2; url)
		support.embed.ly/	image/..	Mozilla/5.0 (compatible; Embedly/0.2; url)
	nb
		www.nb.no/vevfangst	image/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		www.nb.no/vevfangst	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
	weblio
		www.weblio.jp/	text/..	Mozilla/5.0 (compatible; WeblioBot; url)
	topsy
		labs.topsy.com/butterfly/	text/..	Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
	scoutjet
		www.scoutjet.com/	text/..	Mozilla/5.0 (compatible; ScoutJet; url)
	govid
		govid.mobi/bot.php	text/..	Mozilla/5.0 (compatible; gofind; url)
	Anonymouse
		Anonymouse.org/	image/..	url (Unix)
	zapbot
		www.zapbot.org	text/..	Mozilla/5.0 (compatible; ZapBot/0.2o; url)
		www.zapbot.net	text/..	Mozilla/5.0 (compatible; ZapBot/0.2n; url)
		www.zapbot.com	text/..	Mozilla/5.0 (compatible; ZapBot/0.2c; url)
	sentymetr
		sentymetr.pl/bot.html	application/json	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
		sentymetr.pl/bot.html	text/..	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
	netvibes
		www.netvibes.com	text/..	Netvibes (url)
	pinterest
		pinterest.com/	image/..	Pinterest/0.1 url
		pinterest.com/	-	Pinterest/0.1 url
	tumblr
		benderthewebrobot.tumblr.com	text/..	Mozilla/5.0 (compatible; Bender; url)
	grid-son
		grid-son.com	application/json	url
	froute
		labs.froute.jp/pc2m/help.html	text/..	Froute Mobile Gateway/1.0 (url)
	sonyericsson
		www.sonyericsson.com/UAprof/R800xR301.xml	image/..	Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
		www.sonyericsson.com/UAprof/R800xR301.xml	-	Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
	proximic
		www.proximic.com	text/..	Mozilla/5.0 (compatible; proximic; url)
	netarkivet
		netarkivet.dk/website/info.html	image/..	Mozilla/5.0 (compatible; heritrix/1.12.1b url)
		netarkivet.dk/website/info.html	text/..	Mozilla/5.0 (compatible; heritrix/1.12.1b url)
	creativecommons
		wiki.creativecommons.org/Metadata_Scraper	text/..	CC Metadata Scaper url
	instapaper
		www.instapaper.com/	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
	semager
		www.semager.de/blog/semager-bots/	text/..	Mozilla/5.0 (compatible; Semager/1.4c; url)
	mondowindow
		www.mondowindow.com	text/..	MondoWindow (url)
	pagepeeker
		pagepeeker.com/robots	image/..	PagePeeker.com (info: url)
		pagepeeker.com/robots	text/..	PagePeeker.com (info: url)
	rcdtokyo
		www.rcdtokyo.com/pc2m/	text/..	Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
	ibis
		ibis.ne.jp/browser/about.html	-	Mozilla/4.0 (compatible; ibisBrowser; url)
	edu
		ws.nju.edu.cn/falcons/	text/..	Mozilla/5.0 (compatible; Falconsbot; url)
	vbseo
		www.vbseo.com	text/..	Mozilla/4.0 (vBSEO; url)
103,989	total

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	PythonWikipediaBot/1.0
		application/json
		application/xml
		text/..
		-
		image/..
		application/ogg
	GoogleBot-Image/1.0
		text/..
		image/..
		-
	MediaWikiCrawler-Google/2.0 ( mail address )
		text/..
		-
	php wikibot classes
		application/vnd.php.serialized
		text/..
	MoovidaBot/0.1
		text/..
	LinkParser/2.0
		text/..
		-
	GoogleBot-Image/1.0
		text/..
		image/..
		-
		application/vnd.php.serialized
		application/json
	Peachy MediaWiki Bot API Version 1.0
		application/vnd.php.serialized
		-
		text/..
	wikiwix-bot-3.0
		text/..
		-
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
		text/..
		-
		video/ogg
	SemrushBot/0.92
		text/..
		-
		image/..
		application/ogg
		audio/midi
	ClueBot/2.0
		application/vnd.php.serialized
	Answersbot
		text/..
	ClueBot/1.1
		application/vnd.php.serialized
	mail address
		application/vnd.php.serialized
		application/json
		text/..
	Pywikipediabot/2.0
		application/json
	spider
		text/..
		application/vnd.php.serialized
		application/json
		-
		image/..
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
		text/..
		-
		image/..
		application/xml
		application/vnd.php.serialized
	Mozilla 5.0 (Apibot 0.32)
		application/vnd.php.serialized
	gsa-crawler (Enterprise; T2-DS3YYS6PYJWAS; mail address )
		text/..
		-
		image/..
	K-Crawler
		text/..
		-
		application/rsd+xml
		image/..
		application/opensearchdescription+xml
		application/ogg
	wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		-
		text/..
	GSLFbot
		text/..
		-
		image/..
		application/xml
		application/vnd.php.serialized
	AnomieBOT 1.0 (TagDater)
		application/json
		application/x-www-form-urlencoded
	DigitalsmithsBot
		text/..
	BritannicaProjBot mail address
		text/..
	DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		image/..
		application/ogg
	MediaWiki::Bot/3.2.6
		application/json
	DotNetWikiBot/2.97 (Unix 2.6.32.38; )
		text/..
		application/xml
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
		application/json
		application/x-www-form-urlencoded
	Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
		text/..
	bob's crappy crawler; contact: mail address
		text/..
	plantspedia data crawler
		text/..
	python-wikitools/1.2 (User:BernsteinBot)
		application/json
		application/x-www-form-urlencoded
		text/..
	TrueKnowledgeBot bot mail address >
		application/xml
		application/vnd.php.serialized
	Tawbot (public svn release; plwiki)
		text/..
	GoogleBot 2.1
		text/..
		image/..
	Test Webbot
		text/..
	MediaWiki::Bot/1.00
		text/..
		application/json
	DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	MLBot (www.metadatalabs.com/mlbot)
		text/..
		application/vnd.php.serialized
		-
		image/..
	SineBot/1.5.18(User:SineBot)
		application/vnd.php.serialized
		text/..
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		image/..
		text/..
		-
	AnomieBOT 1.0 (OrphanReferenceFixer)
		application/json
	SchoolReviewNetworkWikiBot
		application/json
	FAST Enterprise Crawler 6 used by ESP ( mail address )
		text/..
	Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
		text/..
		-
	AniBot/0.9 php/curl
		application/vnd.php.serialized
		-
	Shad robot/1.0
		text/..
		-
	Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
		text/..
		-
	UCMore Crawler App
		text/..
		-
	JavaCrawler/1.1
		text/..
		-
		image/..
	DotNetWikiBot/2.97 (Unix 5.10.0.0; )
		application/xml
		text/..
	CorenSearchBot/1.5 en libwww-perl/6.02
		text/..
	COIBot/1.00
		text/..
	AnomieBOT 1.0 (FlagIconRemover)
		application/json
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.4.121) Opera Mini/3.1
		image/..
		-
		text/..
	Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
		text/..
		-
	cis455crawler
		text/..
		-
		image/..
	MireoBot
		text/..
		-
	DNSTallyKwBot/0.2
		text/..
	DotNetWikiBot/2.100 (Unix 2.6.32.38; )
		text/..
	TheKeens bot
		text/..
	Webwiki Search Engine Bot - www.webwiki.de
		text/..
	GermCrawler
		application/json
		text/..
	DotNetWikiBot/2.96 (Unix 5.10.0.0; )
		text/..
		application/xml
	SiocWikiBot/1.0
		application/vnd.php.serialized
		text/..
	AnomieBOT 1.0 (TemplateSubster)
		application/json
	AnomieBOT 1.0 (PERTableUpdater)
		application/json
		text/..
	DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	CyberfoxBot
		text/..
		-
		image/..
	OpenSearchServer_Bot
		text/..
	Mozilla/5.0 (compatible; BeetleBot; )
		text/..
		-
		image/..
	wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	XLinkBot/1.00
		text/..
	Twitterbot/1.0
		text/..
		-
		image/..
	SurakWare MediaWiki Bot/1.0
		text/..
		application/xml
	TigrinyaCrawler( mail address )
		text/..
		-
	YBot/0.1
		application/vnd.php.serialized
	HosiryuhosiBot IRC-RecentChanges Util
		text/..
		-
	FAST Enterprise Crawler 6 used by LexisNexis ( mail address )
		text/..
		-
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
		text/..
	DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		application/x-www-form-urlencoded
	HTMLParser/1.6
		text/..
	DotNetWikiBot/2.99 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	AnomieBOT 1.0 (BAGBot)
		application/json
		text/..
	RobBot ( mail address )
		application/vnd.php.serialized
	SearchBot
		text/..
	OrlodrimBot/1.0
		text/..
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
		image/..
		-
		text/..
	HTMLParser/2.0
		text/..
	TVersity Media Robot
		text/..
	HRoestBot, de-wikipedia using pywikipedia framework
		text/..
		application/json
	super cool bot
		application/vnd.php.serialized
	wikbot/1.50 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		-
		text/..
	COIBot/2.0
		text/..
	Bot work. [[no:User:PladaskBot]].
		application/vnd.php.serialized
	GNAA-bot
		text/..
	Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
		application/xml
		text/..
	DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 369 subscribers; feed-id=3852576738117026533)
		application/xml
		-
	AnomieBOT 1.0 (ReplaceExternalLinks2)
		application/json
	FAST Enterprise Crawler/5.3.4 ( mail address )
		text/..
		-
	Freebase Deathbot
		text/..
	wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0
		image/..
		application/json
		-
		text/..
	bitlybot
		text/..
		-
		image/..
	DotNetWikiBot/2.98 (Unix 3.0.0.12; )
		text/..
		application/xml
	wikbotlite/1.50 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		-
		text/..
	MSRBOT
		text/..
	OrangeCrawler/Nutch-1.0 ( mail address )
		text/..
	Goalkeeperbot(User:Beetstra)/1.0
		text/..
	DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	Empedia Bot
		text/..
	Wikibot/1.56 CFNetwork/520.3.2 Darwin/11.3.0 (x86_64) (MacBookPro8,1)
		-
		image/..
		application/json
		text/..
	Markus Möller's super cool bot
		text/..
	WikiCatResearchBot ( mail address )
		text/..
	Local Site Parser 1.0
		text/..
	DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Applied-Technologies-Inc-Spider/Nutch-1.4
		text/..
	AnomieBOT 1.0 (AFDMergeFromCleaner)
		application/json
	My Nutch Spider/Nutch-1.4
		text/..
	AnomieBOT 1.0 (DeletionSortingCleaner)
		application/json
	Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; mail address )
		text/..
		application/pdf
	AnomieBOT 1.0 (RandomPagePicker)
		application/json
	HAZY.SPIDER/Nutch-1.4
		text/..
		-
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
		text/..
		application/xml
	CheMoBot/1.00
		text/..
	MyCuteBot/0.1
		text/..
		application/json
		application/vnd.php.serialized
	Nutch-Spider/Nutch-1.4
		text/..
	AnomieBOT 1.0 (NewArticleAFDTagger)
		application/json
		application/x-www-form-urlencoded
	Mozilla 5.0 (Apibot 0.30b5)
		application/vnd.php.serialized
	ClueBot/2.0 (ClueBot NG Report Interface)
		text/..
	TestCrawler
		text/..
	FUB-RSS-standalone-crawler/Nutch-1.3
		text/..
	HBC Archive Indexerbot 0.9a
		text/..
	cleaner-wikipedia bot / self.maluke.com
		text/..
		application/json
	WikiBot/0.1
		text/..
	wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	gsa-crawler (Enterprise; M2-K3NP42JC6JWAS; mail address )
		text/..
		application/pdf
	Geni ircpybot 1.0
		application/json
		text/..
		application/xml
	DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Zing-BottaBot/1.0
		text/..
	python-wikitools/1.2 (User:LaraBot)
		application/json
	gsa-crawler (Enterprise; M2-PHC4GAAC6LSJA; mail address )
		text/..
	DotNetWikiBot/2.9 (Unix 5.10.0.0; )
		text/..
	Mozilla/5.0 (Bgbot 0.5)
		text/..
	Baiduspider
		text/..
	IssueCrawler
		text/..
	DotNetWikiBot/1.0
		text/..
	GoogleBot
		text/..
		image/..
	rogerbot/1.0
		text/..
	merocrawl/Nutch-1.4 (merobase crawler; www.merobase.com; mail address )
		text/..
		application/pdf
		-
		image/..
	Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
		text/..
		-
	TwynCatBot/0.1 (Contact: www.twyn.com)
		application/json
	FAST Enterprise Crawler/6.7.8 ( mail address )
		text/..
		-
	123peoplebot/1.0
		text/..
	KD Bot
		text/..
		-
		image/..
	MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
		text/..
		application/vnd.php.serialized
	DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
		text/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	deepnet crawler v1.0
		text/..
		application/xml
	gsa-crawler (Enterprise; T3-N9VPLYX3QWSGZ; mail address )
		text/..
18,442	total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, May 25, 2012 16:25
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers