See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 73,289,100 page requests (mime type text/html only!) per day are considered crawler requests, out of 454,144,070 external requests, which is 16.1%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	google
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	-	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortografia4)
		www.google.com/feedfetcher.html	application/xml	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortopedianew)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~cloudcrawling)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/json	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien4)
		code.google.com/p/crawler4j/	text/..	crawler4j (url)
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: rarplayer)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: s~senchaiosrc)
		desktop.google.com/	text/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: s~redconceptual)
		code.google.com/appengine	image/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
		www.google.com/feedfetcher.html	application/xml	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: abdulfat)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: azamasmadi)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pakgalaxy)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: d24-img)
		www.google.com/coop/cse/cref	text/..	FeedFetcher-Google-CoOp; (url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~japantiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kires-roxy)
		code.google.com/appengine	text/..	Mozilla/5.0 AppEngine-Google; (url; appid: s~app3123ak)
		www.google.com/bot.html	image/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	application/json	Mozilla 4.0 AppEngine-Google; (url; appid: prfleme)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: hao1-prxoy)
		code.google.com/appengine	text/..	www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~wikigraph2)
		code.google.com/appengine	text/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
		code.google.com/p/ronda	application/json	Ronda - url
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: worldwide-propaganda)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~senchaiosrc)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~francetiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: mehproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~proxyseekkety)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: simple-tools2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~drizzlprox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tusawebproxy4)
		www.google.com/feedfetcher.html	-	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nagarajhubli-proxy-server)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dkoxyserv)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: 114proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: threewiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: burmansearch)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxyusing121)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kbworld24)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxy-devakishor)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: davrasaurs)
		code.google.com/appengine	application/json	MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ageryder)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: prfleme)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: toom16-10)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webproxy8-2)
		code.google.com/p/ronda	text/..	Ronda - url
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: betafxserver)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ivegotalovelybunch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: thetechnolust)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; documents; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pazvantoff)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wwwwebp2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: python-proxy-server)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: bie99miracle)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: atxproxy)
		code.google.com/appengine	text/..	mail address AppEngine-Google; (url; appid: s~wiki-sherpa)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: boxapp)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: paradigm-web-proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vi-mobile)
		www.google.com/feedfetcher.html	image/..	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline9)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~crowdsurfer100)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlp9)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: adrianswebproxy)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog-test)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web4proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vebproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: openeyeproxy)
		code.google.com/appengine	text/..	Opera/9.80 (Windows NT 6.1; ru) Presto/2.9.168 Version/11.52 AppEngine-Google; (url; appid: s~sib-watchtower)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: chris-homework-helper)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webproxy8-9)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wagagate)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~tpbitalia)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: varlopie)
	facebook
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
		developers.facebook.com	image/..	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.0 (url)
		developers.facebook.com	-	facebookplatform/1.0 (url)
		developers.facebook.com	text/..	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.1 (url)
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible;bingbot/2.0;url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: appcodes)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: howtodoita)
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	application/json	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	application/xml	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider-image(url)
		www.baidu.com/search/spider.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-9)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-8)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-3)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-5)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-4)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-0)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-1)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-2)
		www.baidu.com/search/spider.html	application/xml	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-7)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-6)
		www.baidu.com/search/spider.html	image/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider(url)
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		listing.yahoo.co.jp/support/faq/int/other/other_001.html	text/..	Y!J-BRJ/YATS crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	image/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		developer.yahoo.com/yql/provider	text/..	Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRI/0.0.1 crawler ( url )
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Nano; url)
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRT/1.0 crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	application/vnd.php.serialized	Mozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
		www.yahoo.com/index.html	image/..	Mozilla/5.0 [en] (X11; Linux 2.2.15 i686 url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRW/1.0 crawler (url)
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexDirect/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
		corp.naver.jp/	text/..	Mozilla/5.0 (compatible; NaverJapan/1.0; url)
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	-	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	image/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
	msn
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	image/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-UDiscovery/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-NewsBlogs/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-Products/1.0 (url)
		search.msn.com/msnbot.htm	-	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	-	msnbot/0.01 (url)
		search.msn.com/msnbot.htm	-	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot/0.01 (url)
		search.msn.com/msnbot.htm	image/..	msnbot-NewsBlogs/2.0b (url)
	genieo
		www.genieo.com/webfilter.html	text/..	Mozilla/5.0 (compatible; Genieo/1.0 url)
		www.genieo.com/webfilter.html	application/xml	Mozilla/5.0 (compatible; Genieo/1.0 url)
		www.genieo.com/webfilter.html	image/..	Mozilla/5.0 (compatible; Genieo/1.0 url)
		www.genieo.com/webfilter.html	text/..	Mozilla/5.0 (compatible; Genieo/1.0 url
	blekko
		blekko.com/about/blekkobot	text/..	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
		blekko.com/about/blekkobot	-	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	image/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		toolbar.youdao.com/	image/..	Youdao Toolbar (url)
	soso
		help.soso.com/webspider.htm	text/..	Sosospider(url)
		help.soso.com/webspider.htm	-	Sosospider(url)
		help.soso.com/webspider.htm	application/xml	Sosospider(url)
	cibra
		cibra.de/	text/..	CiBra Data Collector (url)
	sblog
		fulltext.sblog.cz/screenshot/	image/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	-	SeznamBot/3.0 (url)
	www.
		www.	text/..	GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
	echonest
		the.echonest.com/reader/	application/xml	nestReader/0.3 (discovery; url; reader at echonest.com)
		the.echonest.com/reader/	text/..	nestReader/0.3 (discovery; url; reader at echonest.com)
	php
		pear.php.net/	application/vnd.php.serialized	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/0.5.2 (url) PHP/5.2.17
		pear.php.net/	text/..	PEAR HTTP_Request class ( url )
		pear.php.net/	image/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.14
		pear.php.net/package/http_request2	application/xml	HTTP_Request2/2.0.0 (url) PHP/5.3.8
		pear.php.net/	application/xml	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	image/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15
	wwwgogetpapers
		wwwgogetpapers.com/	application/json	User-Agent: GoGetPapersBot (url)
	dataparksearch
		dataparksearch.org/bot	text/..	DataparkSearch/4.54-26052011 (url)
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url)
		www.exabot.com/go/robot	-	Mozilla/5.0 (compatible; Exabot/3.0; url)
	discoveryengine
		discoveryengine.com/discobot.html	text/..	Mozilla/5.0 (compatible; discobot/2.0; url)
		discoveryengine.com/discobot.html	-	Mozilla/5.0 (compatible; discobot/2.0; url)
		discoveryengine.com/discobot.html	image/..	Mozilla/5.0 (compatible; discobot/2.0; url)
	wordpress
		klausgauger.wordpress.com	text/..	WordPress/3.5-alpha; url
		josefboberg.wordpress.com	text/..	WordPress/3.5-alpha; url
		gandalf1316.wordpress.com	text/..	WordPress/3.5-alpha; url
		kterrl.wordpress.com	text/..	WordPress/3.5-alpha-21304; url
		fredvidal.wordpress.com	text/..	WordPress/3.5-alpha; url
		klausgauger.wordpress.com	text/..	WordPress/3.5-alpha-21304; url
		imagenssagradas.wordpress.com	text/..	WordPress/3.5-alpha; url
		support.wordpress.com/contact/	text/..	WordPress.com mShots; url
		gunnyg.wordpress.com	text/..	WordPress/3.5-alpha; url
		kterrl.wordpress.com	text/..	WordPress/3.5-alpha-21273; url
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
		ahrefs.com/robot/	-	Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
		ahrefs.com/robot/	-	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.0 url
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WPCleaner (url)
		en.wikipedia.org	text/..	url
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.18.0 url
		fr.wikipedia.org/wiki/Utilisateur:Salebot	application/json	Salebot, see url (uses Perl MediaWiki::API)
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou Pic Spider/3.0(url)
	bin-co
		www.bin-co.com/php/scripts/load/	text/..	BinGet/1.00.A (url)
		www.bin-co.com/php/scripts/load/	application/vnd.php.serialized	BinGet/1.00.A (url)
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
	yacy
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_33; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-26-generic; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-26-generic; java 1.6.0_24; Indian/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 3.2.0-26-generic; java 1.6.0_24; Indian/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; x86 Windows Server 2008 6.0; java 1.7.0_04; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.34-gentoo-r12; java 1.6.0_33; UTC/en) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_33; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_31; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.1.10-1.16-desktop; java 1.6.0_24; Europe/nl) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_31; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-27-generic; java 1.6.0_24; Indian/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.13-grsec-xxxx-grs-ipv6-64; java 1.6.0_26; America/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; x86 Windows Server 2008 6.0; java 1.6.0_33; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_05; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.1.10-1.13-desktop; java 1.6.0_24; Europe/de) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 3.2.0-27-generic; java 1.6.0_24; Indian/en) url
	numberfound
		www.numberfound.it/	text/..	MyBot/1.0 (url)
		www.numberfound.com/	text/..	MyBot/1.0 (url)
	sistrix
		crawler.sistrix.net/	text/..	Mozilla/5.0 (compatible; SISTRIX Crawler; url)
	toolserver
		wiki.toolserver.org/view/GeoHack	text/..	Geohack (url)
		toolserver.org/~dispenser/	text/..	DispensersTools (url)
		toolserver.org/~dispenser/	application/json	DispensersTools (url)
		toolserver.org/~para/cgi-bin/kmlexport	text/..	url libwww-perl/6.02
	yioop
		www.yioop.com/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot; url)
		www.yioop.com/bot.php	image/..	Mozilla/5.0 (compatible; YioopBot; url)
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url)
		help.zum.com/inquiry	image/..	ZumBot/1.0 (ZUM Search; url)
	okian
		www.okian.ro/	text/..	MyBot/1.0 (url)
	flipboard
		flipboard.com/browserproxy	image/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	application/json	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
		flipboard.com/browserproxy	image/..	null (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	-	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
	daum
		tab.search.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
	wikidict
		www.wikidict.de	text/..	url
	archive-it
		archive-it.org/files/site-owners.html	image/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
		archive-it.org/files/site-owners.html	-	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
		archive-it.org/files/site-owners.html	text/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	image/..	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	-	Mozilla/5.0 (compatible; JikeSpider; url)
	speaktoit
		www.speaktoit.com	application/json	Speaktoit url
	cognarius
		cognarius.com	application/json	AppsArlak/1.0 (url)
		cognarius.com	text/..	AppsArlak/1.0 (url)
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url)
		archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url)
	goo
		help.goo.ne.jp/contact/	text/..	goo wikipedia (url)
		help.goo.ne.jp/door/crawler.html	text/..	ichiro/3.0 (url)
		search.goo.ne.jp/option/use/sub4/sub4-1/	-	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
		search.goo.ne.jp/option/use/sub4/sub4-1/	text/..	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
		help.goo.ne.jp/help/article/1142	text/..	ichiro/3.0 (url)
	traslated
		mymemory.traslated.net/doc/	text/..	Mozilla/5.0 (MyMemory Bot url)
	sf
		liferea.sf.net/	text/..	Liferea/0.x.x (Linux; en_US.UTF-8; url)
		magpierss.sf.net	text/..	MagpieRSS/0.7x (url)
		liferea.sf.net/	text/..	Liferea/1.x.x (Linux; es_ES.UTF-8; url)
	tweetmeme
		tweetmeme.com/	text/..	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
		tweetmeme.com/	-	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
		www.gnip.com/	-	UnwindFetchor/1.0 (url)
	kosmix
		www.kosmix.com/html/kosmos.html	application/xml	Mozilla/5.0(compatible;Kosmos/1.0;url)
		www.kosmix.com/html/kosmos.html	text/..	Mozilla/5.0(compatible;Kosmos/1.0;url)
	github
		github.com/pauldix/typhoeus/tree/master	text/..	Typhoeus - url
		github.com/NeilCrosby/wikislurp	application/vnd.php.serialized	WikiSlurp (url)
		github.com/edsu/wikitweets	application/json	wikitweets <url
	federatedmedia
		federatedmedia.net	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	apercite
		www.apercite.fr/robot/index.html	image/..	Mozilla/5.0 (compatible; Apercite; url)
	accelobot
		www.accelobot.com	text/..	Mozilla/5.0 (compatible; heritrix/1.14.3 url)
	whatrhymeswith
		www.whatrhymeswith.com/site/rhyme-bot	text/..	RhymeBot/0.1 (url)
	newsgator
		www.newsgator.com	text/..	NewsGatorOnline/2.0 (url; 1 subscribers)
		www.newsgator.com/	text/..	FeedDemon/2.7 (url; Microsoft Windows XP)
	avantbrowser
		www.avantbrowser.com	text/..	Avant Browser (url)
		www.avantbrowser.com	text/..	Advanced Browser (url)
	wikiglass
		wikiglass.com	text/..	url : mail address
	feedshow
		www.feedshow.com	text/..	FeedshowOnline (url)
		www.feedshow.com	text/..	Feedshow/x.0 (url; 1 subscriber)
	jetbrains
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 2.0 Release Candidate 1 (url)
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 1.0.x (url)
	emining
		emining.jp/	text/..	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
		emining.jp/	-	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
	enwp
		enwp.org/User:SDPatrolBot	text/..	SDPatrolBot (url)
		enwp.org/User:KingpinBot	text/..	KingpinBot (url)
		enwp.org/User:H3llkn0wz/WikiSharpAPI	text/..	WikiSharpAPI/0.3 url (C# .NET)
	wikimpress
		wikimpress.org/	text/..	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
		wikimpress.org/	-	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
	veveo
		corporate.veveo.net/webmasters.html	text/..	Mozilla/5.0 (compatible; Veveobot; url)
	freebase
		www.freebase.com	text/..	metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
	semager
		www.semager.de/blog/semager-bots/	text/..	Mozilla/5.0 (compatible; Semager/1.4c; url)
	mediawiki
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url)
	netseer
		www.netseer.com/crawler.html	text/..	Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
		www.netseer.com/crawler.html	text/..	Mozilla/5.0 (compatible; Netseer crawler/2.0; url; mail address )
	xbmc
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
	SearchNearMe
		SearchNearMe.com/contact.php	application/vnd.php.serialized	SearchNearMe (url)
		SearchNearMe.com/contact.php	text/..	SearchNearMe (url)
	moviecus
		www.moviecus.com/botcontactinfo.php	application/yaml	moviecus bot (url)
	bibalex
		archive.bibalex.org/bot/	text/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
		archive.bibalex.org/bot/	image/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
	bnf
		www.bnf.fr/fr/outils/a.dl_web_capture_robot.html	image/..	Mozilla/5.0 (compatible; bnf.fr_bot; url)
		www.bnf.fr/fr/outils/a.dl_web_capture_robot.html	text/..	Mozilla/5.0 (compatible; bnf.fr_bot; url)
	paper
		support.paper.li/entries/20023257-what-is-paper-li	text/..	Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
	textdigger
		textdigger.com	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	proximic
		www.proximic.com	text/..	Mozilla/5.0 (compatible; proximic; url)
	drupal
		drupal.org/	text/..	Drupal (url)
		drupal.org/	image/..	Drupal (url)
		drupal.org/	text/..	User-Agent: Drupal (url)
		drupal.org/	-	Drupal (url)
	hatena
		a.hatena.ne.jp/help	text/..	Hatena Antenna/0.5 (url)
	ephorus
		www.ephorus.com/	text/..	Mozilla/5.0 (compatible; Ephorusbot/1.4.4.2; url)
		www.ephorus.com/	text/..	Mozilla/5.0 (compatible; Ephorusbot/1.3.0; url)
	warebay
		www.warebay.com/bot.html	text/..	Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
	it-influentials
		search.it-influentials.com/bot.htm	text/..	Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
	graemef
		graemef.com	text/..	NewsGator FetchLinks extension/0.2.0 (url)
	ponderer
		ponderer.org/download/annotate_google.user.js	text/..	annotate_google; url
	abonti
		www.abonti.com	text/..	Mozilla/5.0 (compatible; Abonti/0.91 - url)
	blogbridge
		www.blogbridge.com/	text/..	BlogBridge 2.13 (url)
	rssreader
		www.rssreader.com	text/..	RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
	seebot
		seebot.org	text/..	Lynx/2.8 (;url)
	winpodder
		winpodder.com	text/..	WinPodder (url)
	nemui
		mozshot.nemui.org/	text/..	Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
	orcabrowser
		www.orcabrowser.com	text/..	Orca Browser (url)
	ranchero
		ranchero.com/netnewswire/	text/..	NetNewsWire/2.x (Mac OS X; url)
	networkedblogs
		www.networkedblogs.com	image/..	NetworkedBlogs (url;) AppEngine-Google; (http://code.google.com/appengine; appid: s~networkedblogshr)
	topsy
		labs.topsy.com/butterfly/	text/..	Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
	zipcommander
		www.zipcommander.com/	text/..	1st ZipCommander (Net) - url
	plagger
		plagger.org/	text/..	Plagger/0.x.xx (url)
	snarfware
		www.snarfware.com/	text/..	Snarfer/0.x.x (url)
	tinyurl
		tinyurl.com/64t5n	text/..	Rome Client (url) Ver: 0.9
	kula
		kula.jp/endo	text/..	endo/1.0 (Mac OS X; ppc i386; url)
	feeds4all
		www.feeds4all.com/feedzcollector	text/..	FeedZcollector v1.x (Platinum) url
	my_website
		my_website.com/my_infopage.html	text/..	Mozilla/5.0 (compatible; heritrix/1.12.1 url)
		my_website.com/my_infopage.html	image/..	Mozilla/5.0 (compatible; heritrix/1.12.1 url)
	grid-son
		grid-son.com	application/json	url
	timewe
		timewe.net	text/..	CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
	rssbandit
		www.rssbandit.org	text/..	RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
	mobileproxy
		mobileproxy.mobi	text/..	Mozilla/5.0 (compatible; MobileSurf; url)
	zootycoon
		www.zootycoon.com	text/..	Zoo Tycoon 2 Client -- url
	trendiction
		www.trendiction.de/bot	text/..	Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
	netnewswireapp
		netnewswireapp.com/mac/	-	NetNewsWire/3.3 (Mac OS X; url; gzip-happy)
		netnewswireapp.com/mac/	-	NetNewsWire/3.3.1 (Mac OS X; url; gzip-happy)
	rockpeaks
		www.rockpeaks.com/contact	text/..	RockPeaks/0.1 (url)
	suggy
		blog.suggy.com/was-ist-suggy/suggy-webcrawler/	text/..	Mozilla/5.0 (compatible; suggybot v0.01a, url)
	embed
		support.embed.ly/	text/..	Mozilla/5.0 (compatible; Embedly/0.2; url)
		support.embed.ly/	image/..	Mozilla/5.0 (compatible; Embedly/0.2; snap; url)
	alexa
		www.alexa.com/site/help/webmasters	text/..	ia_archiver (url; mail address )
	fucinamediale
		labs.fucinamediale.com	text/..	Mozilla/5.0 (compatible; ExperimentalWikiBot/1.0; url)
	searchtechnologies
		www.searchtechnologies.com	text/..	Mozilla/5.0 (compatible; heritrix/1.14.3 url)
	spinn3r
		spinn3r.com/robot	text/..	Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
	tumblr
		benderthewebrobot.tumblr.com	text/..	Mozilla/5.0 (compatible; Bender; url)
	worldaswillandfarce
		worldaswillandfarce.com	text/..	WordPress/3.5-alpha-21304; url
		worldaswillandfarce.com	text/..	WordPress/3.5-alpha-21273; url
	bsurprised
		bsurprised.com/	text/..	BSurprised WikiBox 0.1.3 (url)
	simplepie
		simplepie.org	application/xml	SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
		simplepie.org	text/..	SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
		simplepie.org	text/..	SimplePie/1.2.1 (Feed Parser; url; Allow like Gecko) Build/20111015034325
	dasdonkey
		www.dasdonkey.com	text/..	Mozilla/5.0 (compatible; DonkeyBot/0.1; url)
	weblio
		www.weblio.jp/	text/..	Mozilla/5.0 (compatible; WeblioBot; url)
	nb
		www.nb.no/vevfangst	image/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		www.nb.no/vevfangst	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
	whstour
		whstour.com/osaka	text/..	WordPress/3.3.1; url
		whstour.com/tokyo	text/..	WordPress/3.3.1; url
		whstour.com/nagoya	text/..	WordPress/3.3.1; url
	yunrang
		www.yunrang.com/yrspider.html	text/..	Mozilla/5.0 (compatible; YRSpider; url)
	zapbot
		www.zapbot.com	text/..	Mozilla/5.0 (compatible; ZapBot/0.2c; url)
		www.zapbot.org	text/..	Mozilla/5.0 (compatible; ZapBot/0.2o; url)
		www.zapbot.net	text/..	Mozilla/5.0 (compatible; ZapBot/0.2n; url)
	sentymetr
		sentymetr.pl/bot.html	application/json	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
		sentymetr.pl/bot.html	text/..	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
	fotopedia
		www.fotopedia.com	application/json	Picor (url)
	zookabot
		zookabot.com	text/..	Zookabot/2.4;url
	netvibes
		www.netvibes.com	text/..	Netvibes (url)
	duckduckgo
		duckduckgo.com/duckduckbot.html	text/..	DuckDuckBot/1.1; (url)
		duckduckgo.com/duckduckpreview.html	text/..	DuckDuckPreview/1.0; (url)
		duckduckgo.com/duckduckpreview.html	-	DuckDuckPreview/1.0; (url)
	seokicks
		www.seokicks.de/robot.html	text/..	Mozilla/5.0 (compatible; SEOkicks-Robot url)
	picsearch
		www.picsearch.com/bot.html	text/..	psbot/0.1 (url)
		www.picsearch.com/bot.html	image/..	psbot/0.1 (url)
	kr:6600
		www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp	text/..	url
	pagepeeker
		pagepeeker.com/robots	image/..	PagePeeker.com (info: url)
		pagepeeker.com/robots	text/..	PagePeeker.com (info: url)
	sonyericsson
		www.sonyericsson.com/UAprof/R800xR301.xml	image/..	Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
		www.sonyericsson.com/UAprof/R800xR301.xml	text/..	Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
	Anonymouse
		Anonymouse.org/	image/..	url (Unix)
		Anonymouse.org/	text/..	url (Unix)
	linguee
		www.linguee.com/bot	text/..	Linguee Bot (url; mail address )
	rcdtokyo
		www.rcdtokyo.com/pc2m/	text/..	Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
	localhost
		localhost/~lookformedical/yioop/bot.php	text/..	Mozilla/5.0 (compatible; YIOOP; url)
	superfeedr
		superfeedr.com	application/xml	Superfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
		superfeedr.com	text/..	Superfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
	metamoji
		www.metamoji.com/jp/crawler.html	text/..	Mozilla/5.0 (compatible; MetamojiCrawler/1.0; url
	instapaper
		www.instapaper.com/	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
	plagiarismcheck
		plagiarismcheck.org	application/json	WikiCrawl 1.0b (url contact-mail: mail address )
	edu
		ws.nju.edu.cn/falcons/	text/..	Mozilla/5.0 (compatible; Falconsbot; url)
	rejseliv
		rejseliv.dk	text/..	url (we cache this data for 100 days)
	cdlib
		webarchives.cdlib.org/p/webmasters	image/..	Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 url)
		webarchives.cdlib.org/p/webmasters	text/..	Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 url)
	ibis
		ibis.ne.jp/browser/about.html	image/..	Mozilla/4.0 (compatible; ibisBrowser; url)
	globalspec
		www.globalspec.com/Ocelli	text/..	Ocelli/1.4 (url)
	grapeshot
		www.grapeshot.co.uk/crawler.php	text/..	Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url)
109240.119999995	total

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	PythonWikipediaBot/1.0
		application/json
		application/xml
		text/..
		application/x-www-form-urlencoded
		-
		image/..
		application/pdf
	GoogleBot-Image/1.0
		image/..
		text/..
		-
	php wikibot classes
		application/vnd.php.serialized
		text/..
		-
	MediaWikiCrawler-Google/2.0 ( mail address )
		text/..
		-
	LinkParser/2.0
		text/..
	MoovidaBot/0.1
		text/..
	GoogleBot-Image/1.0
		text/..
		image/..
		-
		application/vnd.php.serialized
		application/json
	badLinks.ru`s crawler v.2
		text/..
		application/xml
	wikiwix-bot-3.0
		text/..
		-
	Peachy MediaWiki Bot API Version 1.0
		application/vnd.php.serialized
	Wikipath Bot (email: mail address )
		application/json
		text/..
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
		text/..
		application/json
		-
	spider
		text/..
		application/vnd.php.serialized
		application/json
		image/..
	Answersbot
		text/..
	Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
		text/..
	ClueBot/1.1
		application/vnd.php.serialized
		text/..
	www.integromedb.org/Crawler
		text/..
		image/..
		-
	Pywikipediabot/2.0
		application/json
	cleaner-wikipedia bot / self.maluke.com
		text/..
		application/json
	Mozilla 5.0 (Apibot 0.32)
		application/vnd.php.serialized
	ClueBot/2.0
		application/vnd.php.serialized
	DigitalsmithsBot
		text/..
	plantspedia data crawler
		text/..
	wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	WikiBot/0.1
		text/..
		application/xml
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		image/..
		text/..
		application/json
		-
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
		text/..
		application/json
		application/vnd.php.serialized
		image/..
		application/xml
	DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		image/..
		application/ogg
	MediaWiki::Bot/3.2.6
		application/json
	mail address
		application/vnd.php.serialized
		text/..
	DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	StatiCrawler/7.0
		text/..
		image/..
	A .NET Web Crawler
		text/..
	AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
		application/json
	python-wikitools/1.2 (User:BernsteinBot)
		application/json
	DotNetWikiBot/2.100 (Unix 2.6.32.38; )
		text/..
	Tawbot (public svn release; plwiki)
		text/..
	MediaWiki::Bot/1.00
		text/..
		application/json
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
		application/ogg
	python2.7.1-wikibot/0.0.2 mail address
		application/json
	Webwiki Search Engine Bot - www.webwiki.de
		text/..
	AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
		application/json
	SineBot/1.5.19(User:SineBot)
		application/vnd.php.serialized
		text/..
	GermCrawler
		application/json
		text/..
	Test Webbot
		text/..
	DotNetWikiBot/2.100 (Unix 5.10.0.0; )
		text/..
		application/xml
	MLBot (www.metadatalabs.com/mlbot)
		text/..
		application/vnd.php.serialized
		-
		image/..
	Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
		text/..
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1)
		application/json
	UCMore Crawler App
		text/..
	YBot/0.1
		application/vnd.php.serialized
	Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
		text/..
	Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
		text/..
	CorenSearchBot/1.7 en libwww-perl/6.04
		text/..
	python-wikitools/1.2 (User:Mr.Z-bot)
		application/json
	AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
		application/json
	Veelobot 0.1
		text/..
		-
		image/..
	CorenSearchBot/1.5 en libwww-perl/6.02
		text/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/x-www-form-urlencoded
		application/xml
	SU Nutch Spider/Nutch-1.4
		text/..
		image/..
	VeeloBot 1.0
		text/..
	wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	SD Crawler/Nutch-0.9 (automated Crawler)
		text/..
	MediaWiki::Bot/3.005002
		application/json
		application/x-www-form-urlencoded
	HTMLParser/1.6
		text/..
		application/x-wiki
		-
	AdMedia bot
		text/..
	Twitterbot/1.0
		text/..
		image/..
		-
	Mozilla/5.0 crawler/suggest.io
		text/..
	AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
		application/json
	GNAA-bot
		text/..
	SiocWikiBot/1.0
		application/vnd.php.serialized
		text/..
	XLinkBot/1.00
		text/..
	OrlodrimBot/1.0
		text/..
	COIBot/1.00
		text/..
	My Crawler/Nutch-1.4
		text/..
	HosiryuhosiBot IRC-RecentChanges Util
		text/..
		-
		application/x-www-form-urlencoded
	mail address mail address – MediaWiki Tcl Bot Framework 0.5
		application/x-www-form-urlencoded
		application/json
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
		image/..
		text/..
		application/json
	MyCuteBot/0.1
		text/..
		application/json
	Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	FAST Search Web Crawler 14.0.0325.0000
		text/..
		-
		application/rsd+xml
		application/xml
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
		text/..
	AniBot/0.9 php/curl
		application/vnd.php.serialized
	useragent: WikiBot/0.1
		text/..
		application/vnd.php.serialized
	SurakWare MediaWiki Bot/1.0
		text/..
		application/xml
	DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	HRoestBot, de-wikipedia using pywikipedia framework
		text/..
		application/json
	HTMLParser/2.0
		text/..
		image/..
	DotNetWikiBot/2.100 (Unix 3.0.0.12; )
		text/..
		application/xml
	AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
		application/json
		text/..
	Phantom.js bot
		image/..
		text/..
	SchoolReviewNetworkWikiBot
		application/json
	TVersity Media Robot
		text/..
	AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
		application/json
		text/..
	bitlybot
		text/..
		image/..
		-
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
		application/json
	infraEnterprise v8 Web Crawler
		-
	uw_transparancy, toolserver.org wikiproject parser
		application/json
		text/..
	DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	WorldOfMusicBetaBot/1.0
		text/..
	wikiparser/1 CFNetwork/454.12.4 Darwin/10.8.0 (x86_64) (MacPro5,1)
		image/..
		text/..
	SearchBot
		text/..
	LauschenBot/1.0 ( mail address )
		text/..
	TwynCatBot/0.1 (Contact: www.twyn.com)
		application/json
	DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	GoogleBot 2.1
		text/..
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.1.0/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 367 subscribers; feed-id=3852576738117026533)
		application/xml
		-
	COIBot/2.0
		text/..
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
		text/..
		application/xml
	Empedia Bot
		text/..
	OrangeCrawler/Nutch-1.0 ( mail address )
		text/..
	Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
		text/..
		-
	Goalkeeperbot(User:Beetstra)/1.0
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.0/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
		application/json
	GoogleBot
		text/..
		image/..
	tellit_rest_bot, contact mail address
		text/..
		application/x-wiki
	SoftNet Nutch Spider/Nutch-1.4
		text/..
		image/..
	robert bot
		text/..
	DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
	QuickFinder Crawler
		text/..
	Zing-BottaBot/2.0
		text/..
	Mozilla 5.0 (Apibot 0.30b5)
		application/vnd.php.serialized
	OdklBot/1.0 ( mail address )
		text/..
		image/..
	wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0
		image/..
		text/..
		application/json
	Geni ircpybot 1.0
		text/..
		application/json
	AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]])
		application/json
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; )
		text/..
		application/xml
	TrueKnowledgeBot bot mail address >
		application/xml
		application/vnd.php.serialized
	LinksCrawler 0.1beta
		text/..
		-
	Mozilla/5.0 (Bgbot 0.5)
		text/..
	SINA_ROBOT; Mozilla/5.0 (Windows; Windows NT 5.1; MSIE8.0; zh-CN; rv:1.9.1.8) Gecko/20100202 Firef8
		text/..
		-
	HBC Archive Indexerbot 0.9a
		text/..
	My Nutch Spider/Nutch-1.4
		text/..
	MediaWiki::Bot 3.1.5
		application/json
	python-wikitools/1.2 (User:LaraBot)
		application/json
	Mozilla/5.0 (iPhone; CPU iPhone OS 4_0_1 like Mac OS X; fr-fr) OrangeBot-Mobile AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8A306 Safari/( mail address )
		image/..
		text/..
		-
	wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0
		image/..
		application/json
		text/..
	Mozilla/5.0 QunarBot/1.0
		text/..
		image/..
	AnomieBOT 1.0 (CHUUClerk; see [[User:AnomieBOT]])
		application/json
		text/..
	AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]])
		application/json
	FTRF: Friendly robot/1.3
		text/..
18639.46	total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, Aug 10, 2012 12:13
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers