See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 76,114,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 525,764,000 external requests, which is 14.5%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	google
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~hr-pulsesubscriber)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~syytacit)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	-	FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/xml	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: rarplayer)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortografia4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortopedianew)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~cloudcrawling)
		desktop.google.com/	text/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/feedfetcher.html	application/json	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: s~redconceptual)
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		www.google.com/feedfetcher.html	application/xml	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien4)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw)
		code.google.com/appengine	image/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~francetiki)
		www.google.com/coop/cse/cref	text/..	FeedFetcher-Google-CoOp; (url)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; documents; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki3)
		code.google.com/p/crawler4j/	text/..	crawler4j (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~wikigraph2)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl)
		code.google.com/appengine	text/..	www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~drizzlprox)
		code.google.com/appengine	text/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: worldwide-propaganda)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~app3123ak)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: mehproxy)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~kasumiremix)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d24-img)
		www.google.com/feedfetcher.html	-	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
		code.google.com/appengine	text/..	Wiki.java 0.26 AppEngine-Google; (url; appid: wikipediatools)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: raja584sekhar)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: usawebproxy0)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: sizzsurf)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vi-mobile)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kires-roxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ivegotalovelybunch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kurizogeorge)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ageryder)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: bubba-ps)
		code.google.com/appengine	application/json	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 KHTML Chrome/22.0.1229.79 Safari/537.4 AppEngine-Google; (url; appid: s~axiachugfw)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki1)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~warawasetekure)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: bypass-filter)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: simple-tools2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tusawebproxy6)
		code.google.com/appengine	image/..	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1 AppEngine-Google; (url; appid: s~enricheddonuts)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: abdulfat)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: kurizogeorge)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web-proxy-hh)
		code.google.com/p/ronda	text/..	Ronda - url
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wmhsonline)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: davrasaurs)
		code.google.com/p/ronda	application/json	Ronda - url
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: burmansearch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pazvantoff)
		code.google.com/appengine	text/..	Mozilla/5.0 AppEngine-Google; (url; appid: s~vodio-app)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tinkernutsearch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~misterhac)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: threewiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: funnyinternetwebsite)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web4proxy)
	facebook
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
		developers.facebook.com	image/..	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	application/json	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.1 (url)
		developers.facebook.com	-	facebookplatform/1.0 (url)
		developers.facebook.com	text/..	facebookplatform/1.0 (url)
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
		www.bing.com/bingbot.htm	application/json	Mozilla/5.0 (compatible; bingbot/2.0; url)
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	GoogleBot/2.1 (url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	application/json	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	application/vnd.php.serialized	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRW/1.0 crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Nano; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	image/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		developer.yahoo.com/yql/provider	text/..	Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRT/1.0 crawler (url)
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	application/json	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider-image(url)
		www.baidu.com/search/spider.html	application/xml	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxy6000)
		www.baidu.com/search/spider.html	image/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexDirect/3.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexNews/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url)
		corp.naver.jp/	text/..	Mozilla/5.0 (compatible; NaverJapan/1.0; url)
	msn
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	image/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-UDiscovery/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-NewsBlogs/2.0b (url)
		search.msn.com/msnbot.htm	-	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-Products/1.0 (url)
		search.msn.com/msnbot.htm	-	msnbot/2.0b (url)
	cibra
		cibra.de/	text/..	CiBra Data Collector (url)
	genieo
		www.genieo.com/webfilter.html	text/..	Mozilla/5.0 (compatible; Genieo/1.0 url)
		www.genieo.com/webfilter.html	application/xml	Mozilla/5.0 (compatible; Genieo/1.0 url)
		www.genieo.com/webfilter.html	image/..	Mozilla/5.0 (compatible; Genieo/1.0 url)
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		toolbar.youdao.com/	image/..	Youdao Toolbar (url)
	php
		pear.php.net/	application/vnd.php.serialized	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/0.5.2 (url) PHP/5.2.17
		pear.php.net/	image/..	PEAR HTTP_Request class ( url )
		pear.php.net/	text/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	application/xml	HTTP_Request2/2.0.0 (url) PHP/5.3.8
		pear.php.net/	application/xml	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	image/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17
	sblog
		fulltext.sblog.cz/screenshot/	image/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	-	SeznamBot/3.0 (url)
		fulltext.sblog.cz/	text/..	SeznamBot/3.0-test (url), I
	toolserver
		wiki.toolserver.org/view/GeoHack	-	Geohack (url)
		wiki.toolserver.org/view/GeoHack	text/..	Geohack (url)
		toolserver.org/~dispenser/	image/..	CacheThumbs/1.2 (url)
		toolserver.org/~dispenser/	-	CacheThumbs/1.2 (url)
		toolserver.org/~guandalug/	application/vnd.php.serialized	Guandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
		toolserver.org/~dispenser/	text/..	DispensersTools (url)
		toolserver.org/~guandalug/	multipart/form-data	- Guandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
		toolserver.org/~para/cgi-bin/kmlexport	text/..	url libwww-perl/6.02
		toolserver.org/~dispenser/	text/..	CacheThumbs/1.2 (url)
		toolserver.org/~platonides/catdown/	image/..	catdown Google_Art_Project (url)
		toolserver.org/~dispenser/	application/x-www-form-urlencoded	DispensersTools (url)
		toolserver.org/~dispenser/	-	DispensersTools (url)
	www.
		www.	text/..	GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
		www.	image/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
	echonest
		the.echonest.com/reader/	application/xml	nestReader/0.3 (discovery; url; reader at echonest.com)
		the.echonest.com/reader/	text/..	nestReader/0.3 (discovery; url; reader at echonest.com)
	wordpress
		tsjok45.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		pilipinasatbp.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		elteologillo.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		josefboberg.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		bibledownloads.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		darcydeedrogorub.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		vicentefatimaeducativo.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		christiannoob.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		ofthehighest.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		speakertoanimals.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		venturajovial.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		02varvara.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		greatriversofhope.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		ambijans.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		lhlukman.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		turismo8f.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		frenyrizq.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		homecomingvets.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		zackytailor.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		gunnyg.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		resepistimewaku.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
		amazingfifa.wordpress.com	text/..	WordPress/3.5-alpha-21535; url
	plos
		alm.plos.org	-	PLoS Article Level Metrics - url
		alm.plos.org	application/json	PLoS Article Level Metrics - url
	soso
		help.soso.com/webspider.htm	text/..	Mozilla/5.0(compatible; Sosospider/2.0; url)
		help.soso.com/webspider.htm	-	Mozilla/5.0(compatible; Sosospider/2.0; url)
		help.soso.com/webspider.htm	application/json	Mozilla/5.0(compatible; Sosospider/2.0; url)
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	-	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	application/json	Mozilla/5.0 (compatible; JikeSpider; url)
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url)
		www.exabot.com/go/robot	-	Mozilla/5.0 (compatible; Exabot/3.0; url)
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.0 url
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WPCleaner (url)
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
		www.majestic12.co.uk/bot.php?	-	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
	249
		173.212.249.18/yioop-v0.90/bot.php	text/..	Mozilla/5.0 (compatible; Woozie! Crawler; url)
		173.212.249.18/bot.php	text/..	Mozilla/5.0 (compatible; Woozie! Crawler; url)
	cognarius
		cognarius.com	-	AppsArlak/1.0 (url)
		cognarius.com	application/json	AppsArlak/1.0 (url)
		cognarius.com	text/..	AppsArlak/1.0 (url)
	bin-co
		www.bin-co.com/php/scripts/load/	application/vnd.php.serialized	BinGet/1.00.A (url)
		www.bin-co.com/php/scripts/load/	text/..	BinGet/1.00.A (url)
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url)
		help.zum.com/inquiry	image/..	ZumBot/1.0 (ZUM Search; url)
	tweetmeme
		tweetmeme.com/	text/..	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
		tweetmeme.com/	text/..	Mozilla/5.0 (compatible; TweetmemeBot/3.0; url)
		tweetmeme.com/	-	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
		tweetmeme.com/	-	Mozilla/5.0 (compatible; TweetmemeBot/3.0; url)
	yacy
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; x86_64 Mac OS X 10.8.2; java 1.6.0_35; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.5.4-2.fc17.x86_64; java 1.7.0_04; America/es) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_05; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-31-generic; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_07; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-042stab061.2; java 1.7.0_07; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-43-generic; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (allip-any; i386 Linux 3.2.0-29-generic-pae; java 1.6.0_24; Europe/en) url
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	application/json	Sogou web spider/4.0(url)
	flipboard
		flipboard.com/browserproxy	image/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	-	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	image/..	null (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	application/json	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
		ahrefs.com/robot/	application/xml	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
		ahrefs.com/robot/	-	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
		ahrefs.com/robot/	application/vnd.php.serialized	Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
	traslated
		mymemory.traslated.net/doc/	text/..	Mozilla/5.0 (MyMemory Bot url)
	daum
		tab.search.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
	coccoc
		help.coccoc.vn/	text/..	coccoc/1.0 (url)
		help.coccoc.vn/	-	coccoc/1.0 (url)
	okian
		www.okian.ro/	text/..	MyBot/1.0 (url)
	dasdonkey
		www.dasdonkey.com	text/..	Mozilla/5.0 (compatible; DonkeyBot/0.1; url)
	enwp
		enwp.org/User:SDPatrolBot	text/..	SDPatrolBot (url)
		enwp.org/User:KingpinBot	text/..	KingpinBot (url)
		enwp.org/User:H3llkn0wz/WikiSharpAPI	text/..	WikiSharpAPI/0.3 url (C# .NET)
	goo
		help.goo.ne.jp/contact/	text/..	goo wikipedia (url)
		goo.gl/7y4SX	text/..	GoogleProducer; (url)
		help.goo.ne.jp/door/crawler.html	text/..	ichiro/3.0 (url)
		search.goo.ne.jp/option/use/sub4/sub4-1/	text/..	ichiro/3.0 (url)
		search.goo.ne.jp/option/use/sub4/sub4-1/	-	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
	discoveryengine
		discoveryengine.com/discoverybot.html	text/..	Mozilla/5.0 (compatible; discoverybot/2.0; url)
		discoveryengine.com/discoverybot.html	-	Mozilla/5.0 (compatible; discoverybot/2.0; url)
	svglib
		svglib.org	image/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		svglib.org	-	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		svglib.org	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
	wikiglass
		wikiglass.com	text/..	url : mail address
	wikidict
		www.wikidict.de	text/..	url
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url)
	ephorus
		www.ephorus.com/	text/..	Mozilla/5.0 (compatible; Ephorusbot/1.4.5.6; url)
	toshiba
		www.toshiba.co.jp/rdc/about/crawl_info.htm	text/..	TosCrawler/Nutch-1.4 (url; ' mail address dot co dot jp')
	netarkivet
		netarkivet.dk/webcrawler/	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		netarkivet.dk/webcrawler/	image/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
	kosmix
		www.kosmix.com/html/kosmos.html	application/xml	Mozilla/5.0(compatible;Kosmos/1.0;url)
	whatrhymeswith
		www.whatrhymeswith.com/site/rhyme-bot	text/..	RhymeBot/0.1 (url)
	wwwgogetpapers
		wwwgogetpapers.com/	application/json	User-Agent: GoGetPapersBot (url)
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
		www.gnip.com/	image/..	UnwindFetchor/1.0 (url)
	federatedmedia
		federatedmedia.net	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	emining
		emining.jp/	text/..	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
	wfcomm
		store.wfcomm.com	text/..	WordPress/3.3.2; url
		blackfridaysnowthrower.wfcomm.com	text/..	WordPress/3.3.2; url
		black-friday-snow-thrower.wfcomm.com	text/..	WordPress/3.3.2; url
	fucinamediale
		labs.fucinamediale.com	text/..	Mozilla/5.0 (compatible; ExperimentalWikiBot/1.0; url)
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url)
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)
		www.archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; archive.org_bot url)
	matuschek
		www.matuschek.net/jobo.html	text/..	JoBo/1.4 (url)
	speaktoit
		www.speaktoit.com	application/json	Speaktoit url
	mediawiki
		www.mediawiki.org/	-	MediaWiki OAI Harvester 0.2 (url)
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url)
	proximic
		www.proximic.com/info/spider.php	text/..	Mozilla/5.0 (compatible; proximic; url)
	sciencecard
		demo.sciencecard.org	application/json	Article Level Metrics - url
	wikimpress
		wikimpress.org/	text/..	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
		wikimpress.org/	-	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
	sf
		liferea.sf.net/	text/..	Liferea/0.x.x (Linux; en_US.UTF-8; url)
		liferea.sf.net/	text/..	Liferea/1.x.x (Linux; es_ES.UTF-8; url)
		magpierss.sf.net	text/..	MagpieRSS/0.7x (url)
		liferea.sf.net/	-	Liferea/1.8.3 (Linux; fr_FR.UTF-8; url)
	apercite
		www.apercite.fr/robot/index.html	image/..	Mozilla/5.0 (compatible; Apercite; url)
	blekko
		blekko.com/about/blekkobot	text/..	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
	drupal
		drupal.org/	image/..	Drupal (url)
		drupal.org/	-	Drupal (url)
		drupal.org/	text/..	Drupal (url)
		drupal.org/	application/xml	Drupal (url)
	semager
		www.semager.de/blog/semager-bots/	text/..	Mozilla/5.0 (compatible; Semager/1.4c; url)
	SearchNearMe
		SearchNearMe.com/contact.php	application/vnd.php.serialized	SearchNearMe (url)
		SearchNearMe.com/contact.php	text/..	SearchNearMe (url)
	moviecus
		www.moviecus.com/botcontactinfo.php	application/yaml	moviecus bot (url)
	picsearch
		www.picsearch.com/bot.html	text/..	psbot/0.1 (url)
		www.picsearch.com/bot.html	image/..	psbot/0.1 (url)
	xbmc
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
		www.xbmc.org	image/..	XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
	nettenis
		www.nettenis.org	text/..	WordPress/3.4.2; url
		www.nettenis.org	image/..	WordPress/3.4.2; url
	bsurprised
		bsurprised.com/	text/..	BSurprised WikiBox 0.1.3 (url)
		bsurprised.com/	-	BSurprised WikiBox 0.1.3 (url)
		bsurprised.com/	application/x-www-form-urlencoded	BSurprised WikiBox 0.1.3 (url)
	easybib
		content.easybib.com/autocite/	application/json	EasyBib AutoCite (url)
		content.easybib.com/autocite/	text/..	EasyBib AutoCite (url)
	alexa
		www.alexa.com/site/help/webmasters	text/..	ia_archiver (url; mail address )
	bibalex
		archive.bibalex.org/bot/	text/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
		archive.bibalex.org/bot/	image/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
	newsgator
		www.newsgator.com/	text/..	FeedDemon/2.7 (url; Microsoft Windows XP)
		www.newsgator.com	text/..	NewsGatorOnline/2.0 (url; 1 subscribers)
	searchtechnologies
		www.searchtechnologies.com	text/..	Mozilla/5.0 (compatible; heritrix/1.14.3 url)
	github
		github.com/pauldix/typhoeus/tree/master	text/..	Typhoeus - url
		github.com/edsu/wikitweets	application/json	wikitweets <url
	hatena
		a.hatena.ne.jp/help	text/..	Hatena Antenna/0.5 (url)
	feedshow
		www.feedshow.com	text/..	Feedshow/x.0 (url; 1 subscriber)
		www.feedshow.com	text/..	FeedshowOnline (url)
	avantbrowser
		www.avantbrowser.com	text/..	Avant Browser (url)
		www.avantbrowser.com	text/..	Advanced Browser (url)
	metamagazine
		metamagazine.com	text/..	WordPress/3.4.2; url
		metamagazine.com	-	WordPress/3.4.2; url
	jetbrains
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 1.0.x (url)
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 2.0 Release Candidate 1 (url)
	uni-potsdam
		www.hpi.uni-potsdam.de/meinel/forschung/web_30/blog_intelligence.html	text/..	HPI-BI-Crawler/0.1(url)/Nutch-2.0-dev
	netnewswireapp
		netnewswireapp.com/mac/	-	NetNewsWire/3.3.2 (Mac OS X; url; gzip-happy)
	zipcode
		zipcode.us	text/..	Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
	textdigger
		textdigger.com	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	plagiarismcheck
		plagiarismcheck.org	application/json	WikiCrawl 1.0b (url contact-mail: mail address )
	coralcdn
		coralcdn.org/	text/..	CoralWebPrx/0.1.20 (See url)
	dentalhygienisttrainingschools
		dentalhygienisttrainingschools.com	text/..	WordPress/3.4.2; url
	a7lambldna
		a7lambldna.com	text/..	WordPress/3.3.1; url
	paper
		support.paper.li/entries/20023257-what-is-paper-li	text/..	Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
	yoursite
		yoursite.com/botinfo	text/..	Mozilla/5.0 (compatible; YourCoolBot/1.0; url)
	trendiction
		www.trendiction.de/bot	text/..	Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
	search
		www.search.ch/rim.html	text/..	UltraSpider3000/1.0 (url)
	comicmix
		www.comicmix.com	text/..	WordPress/3.4.2; url
	sentymetr
		sentymetr.pl/bot.html	text/..	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
		sentymetr.pl/bot.html	application/json	Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
	tumblr
		benderthewebrobot.tumblr.com	text/..	Mozilla/5.0 (compatible; Bender; url)
	feeds4all
		www.feeds4all.com/feedzcollector	text/..	FeedZcollector v1.x (Platinum) url
	rockpeaks
		www.rockpeaks.com/contact	text/..	RockPeaks/0.1 (url)
	turnitin
		www.turnitin.com/robot/crawlerinfo.html	text/..	TurnitinBot/2.1 (url)
	weblio
		www.weblio.jp/	text/..	Mozilla/5.0 (compatible; WeblioBot; url)
	plagger
		plagger.org/	text/..	Plagger/0.x.xx (url)
	simplepie
		simplepie.org	application/xml	SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
	spinn3r
		spinn3r.com/robot	text/..	Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
	winpodder
		winpodder.com	text/..	WinPodder (url)
	nb
		www.nb.no/vevfangst	image/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
		www.nb.no/vevfangst	text/..	Mozilla/5.0 (compatible; heritrix/1.14.4 url)
	kula
		kula.jp/endo	text/..	endo/1.0 (Mac OS X; ppc i386; url)
	ranchero
		ranchero.com/netnewswire/	text/..	NetNewsWire/2.x (Mac OS X; url)
	rcdtokyo
		www.rcdtokyo.com/pc2m/	text/..	Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
	rssreader
		www.rssreader.com	text/..	RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
	ponderer
		ponderer.org/download/annotate_google.user.js	text/..	annotate_google; url
	topsy
		labs.topsy.com/butterfly/	text/..	Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
	timewe
		timewe.net	text/..	CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
	netvibes
		www.netvibes.com	text/..	Netvibes (url)
		www.netvibes.com	image/..	Netvibes (url)
	southmumbairealestate
		southmumbairealestate.me	text/..	WordPress/3.5-alpha-21535; url
	pagepeeker
		pagepeeker.com/robots/	image/..	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 KHTML Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; url
		pagepeeker.com/robots	image/..	PagePeeker.com (info: url)
		pagepeeker.com/robots/	text/..	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 KHTML Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; url
	graemef
		graemef.com	text/..	NewsGator FetchLinks extension/0.2.0 (url)
	seebot
		seebot.org	text/..	Lynx/2.8 (;url)
	salescheapdeals
	instapaper
		www.instapaper.com/	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
	localhost:8888
		localhost:8888	text/..	WordPress/3.4.2; url
	grid-son
		grid-son.com	application/json	url
	tinyurl
		tinyurl.com/64t5n	text/..	Rome Client (url) Ver: 0.9
	nemui
		mozshot.nemui.org/	text/..	Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
	plagsniffer
		plagsniffer.de/	application/json	User-Agent: Plagsniffer (url)
	orcabrowser
		www.orcabrowser.com	text/..	Orca Browser (url)
	zipcommander
		www.zipcommander.com/	text/..	1st ZipCommander (Net) - url
	zeebox
		www.zeebox.com	application/json	Zeebox (url)
		www.zeebox.com	text/..	Zeebox (url)
	Anonymouse
		Anonymouse.org/	image/..	url (Unix)
	gg
		gg.pl/	text/..	GG PeekBot 1.0 ( url http://info.gadu-gadu.pl/praca )
		gg.pl	image/..	GG PeekBot 1.0 (url)
	browsehappy
		browsehappy.com	application/vnd.php.serialized	WordPress/3.5-beta-1; url
	blog
		andrejp21.blog.pl	text/..	WordPress/3; url
	it-influentials
		search.it-influentials.com/bot.htm	text/..	Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
	tineye
		tineye.com/crawler.html	image/..	TinEye/1.1 (url)
	snarfware
		www.snarfware.com/	text/..	Snarfer/0.x.x (url)
	bloople
		wikiwatcher.bloople.net/	-	WikiWatcher url mail address
		wikiwatcher.bloople.net/	text/..	WikiWatcher url mail address
	netseer
		www.netseer.com/crawler.html	text/..	Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
	js-kit
		js-kit.com/	text/..	JS-Kit URL Resolver, url
	stad
		stad.com	text/..	Mozilla/5.0 (compatible; stadbot/1.0; url)
	sencha
		www.sencha.com/products/io/	image/..	Sencha.io-Src; (url)
	rssbandit
		www.rssbandit.org	text/..	RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
	veveo
		corporate.veveo.net/webmasters.html	text/..	Mozilla/5.0 (compatible; Veveobot; url)
	backgroundswitcher
		www.backgroundswitcher.com/	image/..	John's Background Switcher 4.4 (url)
		www.backgroundswitcher.com/	text/..	John's Background Switcher 4.4 (url)
	edu
		ws.nju.edu.cn/falcons/	text/..	Mozilla/5.0 (compatible; Falconsbot; url)
	zootycoon
		www.zootycoon.com	text/..	Zoo Tycoon 2 Client -- url
	blogbridge
		www.blogbridge.com/	text/..	BlogBridge 2.13 (url)
	superfeedr
		superfeedr.com	application/xml	Superfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
	QtWeb
	subshell
		www.subshell.com	text/..	Mozilla/5.0 (compatible; Sophora Linkchecker; url)
	duckduckgo
		duckduckgo.com/duckduckbot.html	text/..	DuckDuckBot/1.1; (url)
	pingdom
		www.pingdom.com/	text/..	Pingdom.com_bot_version_1.4_(url)
	weborking
		weborking.com	text/..	Weborking(url)
		weborking.com	application/json	Weborking(url)
	ibis
		ibis.ne.jp/browser/about.html	image/..	Mozilla/4.0 (compatible; ibisBrowser; url)
	fueto
		fueto.com	application/json	Fueto (url)
	jamaladesigns
		www.jamaladesigns.com	text/..	WordPress/3.4.2; url
118,081	total

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	PythonWikipediaBot/1.0
		application/json
		-
		application/xml
		application/x-www-form-urlencoded
		text/..
	spider
		text/..
		application/vnd.php.serialized
		-
	php wikibot classes
		application/vnd.php.serialized
		-
	GoogleBot-Image/1.0
		image/..
		text/..
		-
	MediaWikiCrawler-Google/2.0 ( mail address )
		text/..
		-
	LinkParser/2.0
		text/..
	Peachy MediaWiki Bot API Version 1.0
		application/vnd.php.serialized
		-
	GoogleBot-Image/1.0
		text/..
		image/..
		-
	Pywikipediabot/2.0
		application/json
		application/x-www-form-urlencoded
	Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
		text/..
		-
		application/json
		image/..
	Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
		text/..
	ClueBot/2.0
		application/vnd.php.serialized
	Answersbot
		text/..
	ClueBot/1.1
		application/vnd.php.serialized
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
		text/..
	AniBot/0.9 php/curl
		application/vnd.php.serialized
	MediaWiki::Bot/3.2.6
		application/json
	Wikipath Bot (email: mail address )
		application/json
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		image/..
		text/..
	unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		-
	python-wikitools/1.2 (User:BernsteinBot)
		application/x-www-form-urlencoded
		application/json
	tigerbot
		text/..
		application/json
	NexiSpider/Nutch-1.5.1
		text/..
	plantspedia data crawler
		text/..
	Mozilla 5.0 (Apibot 0.32)
		application/vnd.php.serialized
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
		text/..
		-
	DigitalsmithsBot
		text/..
	DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	wikbot/1.60 CFNetwork/609 Darwin/13.0.0
		image/..
		application/json
		text/..
		-
	AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
		application/json
	Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl
		text/..
	mail address
		application/vnd.php.serialized
	DotNetWikiBot/2.100 (Unix 2.6.32.38; )
		text/..
	Tawbot (public svn release; plwiki)
		text/..
	Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
		text/..
	SineBot/1.5.19(User:SineBot)
		application/vnd.php.serialized
		text/..
	DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		image/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; )
		text/..
		application/xml
	AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
		application/json
	AdMedia bot
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	GermCrawler
		application/json
	Webwiki Search Engine Bot - www.webwiki.de
		text/..
	TMFBot
		text/..
	DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	MediaWiki::Bot/3.005002
		application/json
	Test Webbot
		text/..
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
		-
		text/..
	Bot
		text/..
	JavaCrawler/1.1
		text/..
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
		image/..
		text/..
	HRoestBot, de-wikipedia using pywikipedia framework
		application/x-www-form-urlencoded
		text/..
		application/json
	AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
		application/json
		text/..
	SiocWikiBot/1.0
		application/vnd.php.serialized
		text/..
	wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		text/..
		application/json
	HosiryuhosiBot IRC-RecentChanges Checker
		text/..
		application/x-www-form-urlencoded
	www.integromedb.org/Crawler
		text/..
	DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	DotNetWikiBot/2.100 (Unix 5.10.0.0; )
		text/..
		application/xml
	OrlodrimBot/1.0
		text/..
		-
		application/x-www-form-urlencoded
	mySpider/Nutch-1.5.1
		text/..
		-
	Twitterbot/1.0
		text/..
		image/..
	DotNetWikiBot/2.100 (Unix 3.0.0.12; )
		text/..
		application/xml
	TVersity Media Robot
		text/..
	CorenSearchBot/1.7 en libwww-perl/6.02
		text/..
	- Peachy MediaWiki Bot API Version 1.0
		multipart/form-data
	cleaner-wikipedia bot / self.maluke.com
		application/json
	SurakWare MediaWiki Bot/1.0
		text/..
	wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		application/json
		image/..
	COIBot/1.00
		text/..
	Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
		text/..
	DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1)
		application/x-www-form-urlencoded
		application/json
	TheKeens bot
		text/..
	microbot
		text/..
	Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
		text/..
	Mozilla/5.0 (compatible; Tbot/1.0;)
		text/..
	Peachy MediaWiki Bot API Version 0.1beta
		application/vnd.php.serialized
		-
	AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
		application/json
	UCMore Crawler App
		text/..
		-
	wikiwix-bot-3.0
		text/..
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
		text/..
		application/xml
	YBot/0.1
		application/vnd.php.serialized
	LauschenBot/1.0 ( mail address )
		text/..
	Mozilla/5.0 (Bgbot 0.5)
		text/..
	gsa-crawler test (Enterprise; T3-N9HAEEX39WSGH; mail address )
		text/..
		-
	Phantom.js bot
		image/..
	AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
		application/json
		text/..
	Mozilla/5.0 (compatible; UnisterBot; mail address )
		text/..
	mail address mail address – MediaWiki Tcl Bot Framework 0.5
		application/x-www-form-urlencoded
		application/json
	- php wikibot classes
		multipart/form-data
	ClueBot/2.0 (ClueBot NG Report Interface)
		text/..
	TwynCatBot/0.1 (Contact: www.twyn.com)
		application/json
	Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
		text/..
		-
	EasouSpider
		text/..
	EarwigBot/0.2.dev.git4ff7612a (Python/2.7.3; https://github.com/earwig/earwigbot; mail address )
		application/json
		application/x-www-form-urlencoded
	AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
		application/json
	Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
		-
		image/..
		text/..
	Goalkeeperbot(User:Beetstra)/1.0
		text/..
	wikbotlite/1.60 CFNetwork/609 Darwin/13.0.0
		image/..
		application/json
		text/..
	Local Site Parser 1.0
		text/..
	XLinkBot/1.00
		text/..
	HTMLParser/2.0
		text/..
	bitlybot
		text/..
	FAST Search Web Crawler 14.0.0325.0000
		text/..
	Handelabra WikiBot
		application/vnd.php.serialized
	Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 389 subscribers; feed-id=3852576738117026533)
		application/xml
		-
	infraEnterprise v8 Web Crawler
		-
	MyCuteBot/0.1
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.1.0/1.8.5.168;) Opera Mini/3.1
		image/..
	KumulBot/0.24
		application/vnd.php.serialized
	Metabot 0.1
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.0/1.8.5.168;) Opera Mini/3.1
		text/..
		image/..
	MediaWiki::Bot 3.1.5
		application/json
		application/x-www-form-urlencoded
	robert bot
		text/..
	SchoolReviewNetworkWikiBot
		application/json
	PopScreenBot
		text/..
	AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
		application/json
	BibBot/0.9 (urshofer.ch)
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.2/1.8.5.168;) Opera Mini/3.1
		image/..
		-
	Erel Bot
		text/..
	Mozilla 5.0 (Apibot 0.30b5)
		application/vnd.php.serialized
	My Nutch Spider/Nutch-1.5.1
		text/..
	SoxBot PHP
		-
		application/vnd.php.serialized
	My Nutch Spider/Nutch-1.4
		text/..
	wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0
		image/..
	WorldOfMusicBetaBot/1.0
		text/..
	Explorer Bot
		text/..
	SiteSeekerCrawler/1.0
		text/..
	Tweetbot/25100 CFNetwork/609 Darwin/13.0.0
		image/..
	GoogleBot
		text/..
	JaGaBot (using SxWiki 0.1.0)
		application/x-www-form-urlencoded
	MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
		text/..
	TweetbotPad/25000 CFNetwork/609 Darwin/13.0.0
		text/..
		image/..
	DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	AnomieBOT 1.0 (CHUUClerk; see [[User:AnomieBOT]])
		application/json
	python-wikitools/1.2 (User:LaraBot)
		application/json
		application/x-www-form-urlencoded
	FAST Enterprise Crawler 6 used by MS ( mail address )
		text/..
	COMODOSpider/Nutch-1.2
		text/..
	COIBot/2.0
		text/..
	Rainmeter WebParser plugin
		application/xml
	UniversalFeedParser/5.1.1 https://code.google.com/p/feedparser/
		application/xml
	FAST Enterprise Crawler 6 used by MS-FAST ( mail address )
		-
		text/..
	- Peachy MediaWiki Bot API Version 0.1beta
		multipart/form-data
	SearchBot
		text/..
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.0.3/1.8.5.168;) Opera Mini/3.1
		text/..
		image/..
	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.8.1; maxamine.com-robot) Gecko/20061010 Firefox/2.0
		text/..
	MediaWiki::Bot/3.1.6 (User:SporkBot)
		application/json
38,856	total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Tue, Dec 4, 2012 19:57
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers