Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 9 Jan 2013 - 30 Jan 2013 (last 12 months)

This analysis is based on a 1:1000 sampled server log (squids)

Other reports:
Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

Notes on reliability of these data

Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot
For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'
In total 104,812,910 page requests (mime type text/html only!) per day are considered crawler requests, out of 605,947,230 external requests, which is 17.3%

Page requests for crawlers that specify a url in the agent string

Count
x 1000 Secondary domain
(~site) name URL Mime type User agent

google

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/bot.html - Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/bot.html text/.. Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -

www.google.com/bot.html - Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -

desktop.google.com/ application/xml Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -

www.google.com/bot.html image/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/feedfetcher.html image/.. Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -

www.google.com/bot.html text/.. SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -

www.google.com/bot.html - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -

desktop.google.com/ image/.. Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -

desktop.google.com/ - Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -

code.google.com/p/crawler4j/ text/.. crawler4j (url) - -

www.google.com/feedfetcher.html - FeedFetcher-Google; (url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ortografia4) - -

www.google.com/feedfetcher.html application/xml FeedFetcher-Google; (url) - -

code.google.com/appengine application/json AppEngine-Google; (url; appid: s~redconceptual) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ortopedianew) - -

code.google.com/appengine text/.. WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -

www.google.com/feedfetcher.html text/.. Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wikien3) - -

code.google.com/appengine text/.. www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing) - -

www.google.com/feedfetcher.html application/json Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -

www.google.com/feedfetcher.html text/.. FeedFetcher-Google; (url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: rarplayer) - -

desktop.google.com/ text/.. Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -

www.google.com/bot.html text/.. DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -

code.google.com/appengine text/.. Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -

www.google.com/feedfetcher.html application/xml Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wikien4) - -

code.google.com/appengine application/xml AppEngine-Google; (url; appid: wikipedia-raw) - -

www.google.com/bot.html - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -

docs.google.com image/.. Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -

desktop.google.com/ - Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -

code.google.com/appengine text/.. Wiki.java 0.27 AppEngine-Google; (url; appid: wikipediatools) - -

www.google.com/bot.html text/.. GoogleBot/2.1 (url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~kasumiremix) - -

docs.google.com image/.. Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -

code.google.com/appengine application/json Mozilla/5.0 AppEngine-Google; (url; appid: s~redconceptual) - -

code.google.com/appengine text/.. Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -

code.google.com/appengine - Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -

code.google.com/appengine text/.. Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -

code.google.com/appengine text/.. Mozilla/5.0 AppEngine-Google; (url; appid: s~birthday-stats) - -

code.google.com/appengine image/.. AppEngine-Google; (url; appid: d24-img) - -

www.google.com/bot.html NONE/wikipedia - Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/coop/cse/cref text/.. FeedFetcher-Google-CoOp; (url) - -

www.google.com/feedfetcher.html text/.. Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -

desktop.google.com/ application/xml Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: usawebdl) - -

code.google.com/appengine text/.. route-hacker alpha AppEngine-Google; (url; appid: s~routehackers) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~francetiki) - -

code.google.com/appengine image/.. AppEngine-Google; (url; appid: usawebproxy0) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: d24-img) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: boxapp) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~japantiki) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~app3123ak) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~app-cruxbot) - -

www.google.com/feedfetcher.html - Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: worldwide-propaganda) - -

code.google.com/appengine image/.. Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: threewiki) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~hr-pulsesubscriber) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~drizzlprox) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~theunblock) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: pakgalaxy) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kires-roxy) - -

code.google.com/appengine application/json MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~liquid-helium) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~espanatiki) - -

code.google.com/appengine image/.. AppEngine-Google; (url; appid: boxapp) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: mehproxy) - -

desktop.google.com/ image/.. Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -

www.google.com/bot.html application/ogg Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxyusing121) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: your-zone) - -

www.google.com/feedfetcher.html text/.. FeedFetcher-Google; (url) ja,en-us;q=0.7,en;q=0.3 -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~proxyseekkety) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~italiatiki) - -

docs.google.com text/.. Mozilla/5.0 (compatible; GoogleDocs; script; url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tunisistan) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: python-proxy-server) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: atxproxy) - -

code.google.com/appengine image/.. AppEngine-Google; (url; appid: azshaderserver) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: my-api) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: toom16-10) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: thetechnolust) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: azshaderserver) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: usawebproxy0) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ridemyhell) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: davrasaurs) - -

code.google.com/appengine text/.. Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: vebproxy) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: dkoxyserv) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~keytanwiki3) - -

www.google.com/feedfetcher.html image/.. FeedFetcher-Google; (url) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: varlopie) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxy-devakishor) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~stremor-crawler) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: vadim-proxy) - -

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxy12345) - -

facebook

www.facebook.com/externalhit_uatext.php text/.. facebookexternalhit/1.1 (url) - -

www.facebook.com/externalhit_uatext.php image/.. facebookexternalhit/1.1 (url) - -

www.facebook.com/externalhit_uatext.php text/.. facebookexternalhit/1.0 (url) - -

www.facebook.com/externalhit_uatext.php image/.. facebookexternalhit/1.0 (url) - -

www.facebook.com/externalhit_uatext.php - facebookexternalhit/1.1 (url) - -

developers.facebook.com image/.. facebookplatform/1.0 (url) - -

www.facebook.com/externalhit_uatext.php - facebookexternalhit/1.0 (url) - -

developers.facebook.com - facebookplatform/1.0 (url) - -

bing

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) - -

www.bing.com/bingbot.htm - Mozilla/5.0 (compatible; bingbot/2.0; url) - -

www.bing.com/bingbot.htm - Mozilla/5.0 (compatible; bingbot/2.0; url) -

www.bing.com/bingbot.htm image/.. Mozilla/5.0 (compatible; bingbot/2.0; url) - -

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) en -

www.bing.com/bingbot.htm application/vnd.php.serialized Mozilla/5.0 (compatible; bingbot/2.0; url) - -

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5 - -

www.bing.com/bingbot.htm application/json Mozilla/5.0 (compatible; bingbot/2.0; url) - -

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) -

www.bing.com/bingbot.htm - Mozilla/5.0 (compatible; bingbot/2.0; url) en -

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3 - -

www.bing.com/bingbot.htm application/xml Mozilla/5.0 (compatible; bingbot/2.0; url) - -

www.bing.com/bingbot.htm application/ogg Mozilla/5.0 (compatible; bingbot/2.0; url) - -

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8) - -

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4) - -

www.bing.com/bingbot.htm image/.. Mozilla/5.0 (compatible; bingbot/2.0; url) -

google?

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/bot.html - Mozilla/5.0 (compatible; GoogleBot/2.1; url) -

www.google.com/bot.html text/.. GoogleBot/2.1 (url) - -

www.google.com/bot.html - Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/bot.html image/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/bot.html text/.. GoogleBot/2.1 (url) en-us,en;q=0.5 -

www.google.com/bot.html application/vnd.php.serialized Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/bot.html application/json Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/bot.html text/.. GoogleBot/2.1 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -

www.google.com/bot.html - Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -

www.google.com/bot.html - GoogleBot/2.1 (url) en-us,en;q=0.5

www.google.com/bot.html application/xml Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

www.google.com/bot.html text/.. Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -

www.google.com/bot.html - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -

www.google.com/bot.html - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -

yahoo

help.yahoo.com/help/us/ysearch/slurp image/.. Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -

help.yahoo.com/help/us/ysearch/slurp application/json Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. Y!J-BRJ/YATS crawler (url) - -

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html image/.. 'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -

help.yahoo.com/help/us/ysearch/slurp - Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. 'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -

developer.yahoo.com/yql/provider text/.. Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -

help.yahoo.com/help/us/ysearch/slurp - Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. Y!J-BRT/1.0 crawler (url) - -

help.yahoo.com/help/us/ysearch/slurp application/xml Mozilla/5.0 (compatible; Yahoo! Slurp;url) - -

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. Y!J-BRW/1.0 crawler (url) ja, *;q=0.5 -

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. Y!J-BRW/1.0 crawler (url) - -

yandex

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -

yandex.com/bots - Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -

yandex.com/bots - Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexBot/3.0; url) - -

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexImages/3.0; url) - -

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexImages/3.0; url) - -

yandex.com/bots - Mozilla/5.0 (compatible; YandexBot/3.0; url) -

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -

yandex.com/bots - Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -

yandex.com/bots application/ogg Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -

yandex.com/bots application/json Mozilla/5.0 (compatible; YandexBot/3.0; url) - -

baidu

www.baidu.com/search/spider.html text/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -

www.baidu.com/search/spider.html text/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -

www.baidu.com/search/spider.html text/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -

www.baidu.com/search/spider.html text/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -

www.baidu.com/search/spider.html - Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -

www.baidu.com/search/spider.html - Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -

www.baidu.com/search/spider.html - Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -

www.baidu.com/search/spider.htm image/.. Baiduspider-image(url) - -

www.baidu.com/search/spider.html text/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -

www.baidu.com/search/spider.htm text/.. Baiduspider-image(url) - -

www.baidu.com/search/spider.html - Mozilla/5.0 (compatible; Baiduspider/2.0; url) -

www.baidu.com/search/spider.html application/json Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -

www.baidu.com/search/spider.html application/json Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -

www.baidu.com/search/spider.html - Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -

www.baidu.com/search/spider.html application/xml Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -

msn

search.msn.com/msnbot.htm text/.. msnbot-NewsBlogs/2.0b (url) - -

search.msn.com/msnbot.htm text/.. msnbot-media/1.1 (url) - -

search.msn.com/msnbot.htm text/.. msnbot/2.0b (url) - -

search.msn.com/msnbot.htm image/.. msnbot-media/1.1 (url) - -

search.msn.com/msnbot.htm image/.. msnbot-NewsBlogs/2.0b (url) - -

search.msn.com/msnbot.htm text/.. msnbot-Products/1.0 (url) - -

search.msn.com/msnbot.htm image/.. msnbot/2.0b (url) - -

search.msn.com/msnbot.htm - msnbot-media/1.1 (url) - -

search.msn.com/msnbot.htm text/.. msnbot-UDiscovery/2.0b (url) - -

search.msn.com/msnbot.htm text/.. msnbot/0.01 (url) - -

naver

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -

help.naver.com/robots/ - Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -

help.naver.com/robots/ - Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) - -

help.naver.com/robots/ image/.. Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -

help.naver.com/robots/ image/.. Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -

help.naver.com/robots/ image/.. Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -

help.naver.com/robots/ text/.. Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) tr,en;q=0.5 -

help.naver.com/robots/ application/json Yeti/1.0 (NHN Corp.; url) - -

cibra

cibra.de/ text/.. CiBra Data Collector (url) - -

youdao

www.youdao.com/help/webmaster/spider/ text/.. Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -

www.youdao.com/help/webmaster/spider/ text/.. Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -

www.youdao.com/help/webmaster/spider/ - Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -

www.youdao.com/help/webmaster/spider/ - Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -

www.youdao.com/help/reader/faq/topic006/ - Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;url;1 subscribers;) - -

genieo

www.genieo.com/webfilter.html text/.. Mozilla/5.0 (compatible; Genieo/1.0 url) - -

www.genieo.com/webfilter.html application/xml Mozilla/5.0 (compatible; Genieo/1.0 url) - -

www.genieo.com/webfilter.html image/.. Mozilla/5.0 (compatible; Genieo/1.0 url) - -

www.genieo.com/webfilter.html image/.. Mozilla/5.0 (compatible; Genieo/1.0 url) en,*

sblog

fulltext.sblog.cz/screenshot/ image/.. Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -

fulltext.sblog.cz/ text/.. SeznamBot/3.0 (url) cs -

fulltext.sblog.cz/screenshot/ text/.. Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -

fulltext.sblog.cz/ - SeznamBot/3.0 (url) cs -

fulltext.sblog.cz/screenshot/ - Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -

fulltext.sblog.cz/screenshot/ text/.. Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -

fulltext.sblog.cz/screenshot/ - Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -

80legs

www.80legs.com/webcrawler.html text/.. Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -

soso

help.soso.com/webspider.htm text/.. Mozilla/5.0(compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -

help.soso.com/webspider.htm text/.. Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -

help.soso.com/webspider.htm - Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -

help.soso.com/webspider.htm image/.. Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -

help.soso.com/webspider.htm text/.. Mozilla/5.0(compatible; Sosospider/2.0; url) - -

help.soso.com/webspider.htm text/.. Sosospider(url) - -

ahrefs

ahrefs.com/robot/ text/.. Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -

ahrefs.com/robot/ - Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -

ahrefs.com/robot/ - Mozilla/5.0 (compatible; AhrefsBot/4.0; url) -

ahrefs.com/robot/ application/json Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -

zum

help.zum.com/inquiry text/.. ZumBot/1.0 (ZUM Search; url) - -

help.zum.com/inquiry image/.. ZumBot/1.0 (ZUM Search; url) - -

blekko

blekko.com/about/blekkobot text/.. Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -

blekko.com/about/blekkobot - Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -

php

pear.php.net/ application/vnd.php.serialized PEAR HTTP_Request class ( url ) - -

pear.php.net/package/http_request2 text/.. HTTP_Request2/0.5.2 (url) PHP/5.2.17 - -

pear.php.net/ image/.. PEAR HTTP_Request class ( url ) - -

pear.php.net/ text/.. PEAR HTTP_Request class ( url ) - -

pear.php.net/package/http_request2 application/xml HTTP_Request2/2.0.0 (url) PHP/5.3.8 - -

pear.php.net/ application/xml PEAR HTTP_Request class ( url ) - -

pear.php.net/package/http_request2 text/.. HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17 - -

pear.php.net/package/http_request2 image/.. HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15 - -

pear.php.net/package/http_request2 image/.. HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -

exabot

www.exabot.com/go/robot text/.. Mozilla/5.0 (compatible; Exabot/3.0; url) - -

www.exabot.com/go/robot text/.. Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -

www.exabot.com/go/robot - Mozilla/5.0 (compatible; Exabot/3.0; url) - -

www.exabot.com/go/robot - Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -

143

173.13.143.74/bot.php text/.. Mozilla/5.0 (compatible; YioopBot; url) - -

173.13.143.74/bot.php - Mozilla/5.0 (compatible; YioopBot; url) - -

173.13.143.74/bot.php image/.. Mozilla/5.0 (compatible; YioopBot; url) - -

echonest

the.echonest.com/reader/ application/xml nestReader/0.3 (discovery; url; reader at echonest.com) en -

the.echonest.com/reader/ text/.. nestReader/0.3 (discovery; url; reader at echonest.com) en -

toolserver

wiki.toolserver.org/view/GeoHack text/.. Geohack (url) - -

wiki.toolserver.org/view/GeoHack - Geohack (url) -

toolserver.org/~dispenser/ - CacheThumbs/1.2 (url) -

toolserver.org/~dispenser/ image/.. CacheThumbs/1.2 (url) - -

toolserver.org/~dispenser/ text/.. CacheThumbs/1.2 (url) - -

toolserver.org/~dispenser/ text/.. DispensersTools (url) - -

toolserver.org/~dispenser/ - DispensersTools (url) -

toolserver.org/~dispenser/ application/x-www-form-urlencoded DispensersTools (url) -

toolserver.org/~para/cgi-bin/kmlexport text/.. url libwww-perl/6.02 - -

toolserver.org/~dispenser/ application/json DispensersTools (url) - -

wordpress

josefboberg.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

tsjok45.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

kabarislam.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

greatriversofhope.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

jaggedykaye.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

lesliebrodie.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

beyondthelies.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

gunnyg.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

ambiya999dotcom.wordpress.com text/.. WordPress/3.6-alpha-23288; url - -

www.

www. text/.. GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -

www. text/.. GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -

www. text/.. GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -

wikipedia

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.20.0 url - -

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.19.0 url - -

en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation text/.. WPCleaner (url) - -

en.wikipedia.org/wiki/Wikipedia:Huggle - Huggle/2.1.19.0 url -

fr.wikipedia.org/wiki/Utilisateur:Salebot application/json Salebot, see url (uses Perl MediaWiki::API) - -

sk.wikipedia.org/wiki/Redaktor:TeslaBot - TeslaBot (url) - -

sogou

www.sogou.com/docs/help/webmasters.htm#07 text/.. Sogou web spider/4.0(url) zh-cn -

www.sogou.com/docs/help/webmasters.htm#07 - Sogou web spider/4.0(url) zh-cn -

www.sogou.com/docs/help/webmasters.htm#07 text/.. Sogou web spider/4.0(url) - -

www.sogou.com/docs/help/webmasters.htm#07 application/json Sogou web spider/4.0(url) - -

www.sogou.com/docs/help/webmasters.htm#07 image/.. Sogou Pic Spider/3.0(url) zh-cn -

finecomb

finecomb.com/ - api/1.1 (url; mail address ) - -

finecomb.com/ application/json api/1.1 (url; mail address ) - -

wikidict

www.wikidict.de text/.. url - -

wwwgogetpapers

wwwgogetpapers.com/ application/json User-Agent: GoGetPapersBot (url) - -

traslated

mymemory.traslated.net/doc/ text/.. Mozilla/5.0 (MyMemory Bot url) - -

majestic12

www.majestic12.co.uk/bot.php? text/.. Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -

www.majestic12.co.uk/bot.php? text/.. Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -

www.majestic12.co.uk/bot.php? - Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) -

jike

shoulu.jike.com/spider.html text/.. Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -

shoulu.jike.com/spider.html text/.. Mozilla/5.0 (compatible; JikeSpider; url) - -

shoulu.jike.com/spider.html - Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -

shoulu.jike.com/spider.html image/.. Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -

zookabot

zookabot.com text/.. Zookabot/2.5;url - -

zookabot.com image/.. Zookabot/2.5;url - -

coccoc

help.coccoc.vn/ text/.. coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -

help.coccoc.vn/ - coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -

help.coccoc.vn/ text/.. coccoc/1.0 (url) - -

help.coccoc.vn/ text/.. coccoc/1.0 (url) en-us,en;q=0.5 -

flipboard

flipboard.com/browserproxy image/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -

flipboard.com/browserproxy text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -

flipboard.com/browserproxy text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -

flipboard.com/browserproxy application/json Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url) en-us,en;q=0.5 -

flipboard.com/browserproxy image/.. null (FlipboardProxy/1.1; url) - -

flipboard.com/browserproxy - Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -

flipboard.com/browserproxy - Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -

yacy

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html - yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 FreeBSD 9.1-RELEASE; java 1.7.0_011; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.6.11-1-ARCH; java 1.7.0_03; Europe/fr) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.6.11-1-ARCH; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html - yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_24; Europe/fr) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; x86_64 Mac OS X 10.8.1; java 1.6.0_37; Asia/ru) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.7.4-1-ARCH; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_07; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (torworld/any; amd64 Linux 3.4.0-5-generic; java 1.7.0_09; Europe/en) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (webportal/global; x86 Windows 7 6.1; java 1.6.0_18; Europe/fr) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (webportal-global; x86 Windows XP 5.1; java 1.7.0_07; Europe/de) url en-us,en;q=0.5 -

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.5.0-21-generic; java 1.7.0_09; America/en) url en-us,en;q=0.5 -

yacy.net/bot.html - yacybot (torworld/any; amd64 Linux 3.4.0-5-generic; java 1.7.0_09; Europe/en) url en-us,en;q=0.5 -

SearchNearMe

SearchNearMe.com/contact.php application/vnd.php.serialized SearchNearMe (url) - -

SearchNearMe.com/contact.php text/.. SearchNearMe (url) - -

elcidharth

elcidharth.com text/.. WordPress/3.6-alpha-23288; url - -

elcidharth.com image/.. WordPress/3.6-alpha-23288; url - -

ac

www.tkl.iis.u-tokyo.ac.jp/~crawler/ text/.. Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -

www.clips.ua.ac.be/pages/pattern application/json Pattern/2.3 url - -

www.tkl.iis.u-tokyo.ac.jp/~crawler/ - Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -

www.ninjal.ac.jp/corpus_center/ulc/crawl-en text/.. Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -

www.tkl.iis.u-tokyo.ac.jp/~crawler/ text/.. Mozilla/5.0 (compatible; Steeler/3.5; url) - -

www.tkl.iis.u-tokyo.ac.jp/~crawler/ image/.. Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -

cognarius

cognarius.com - AppsArlak/1.0 (url) -

cognarius.com application/json AppsArlak/1.0 (url) - -

cognarius.com text/.. AppsArlak/1.0 (url) - -

bin-co

www.bin-co.com/php/scripts/load/ text/.. BinGet/1.00.A (url) - -

www.bin-co.com/php/scripts/load/ application/vnd.php.serialized BinGet/1.00.A (url) - -

goo

help.goo.ne.jp/contact/ text/.. goo wikipedia (url) - -

goo.gl/7y4SX text/.. GoogleProducer; (url) - -

search.goo.ne.jp/option/use/sub4/sub4-1/ text/.. ichiro/3.0 (url) - -

help.goo.ne.jp/door/crawler.html text/.. ichiro/3.0 (url) - -

search.goo.ne.jp/option/use/sub4/sub4-1/ - DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -

search.goo.ne.jp/option/use/sub4/sub4-1/ text/.. DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -

goo.gl/7y4SX image/.. GoogleProducer; (url) - -

search.goo.ne.jp/option/use/sub4/sub4-1/ - DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -

search.goo.ne.jp/option/use/sub4/sub4-1/ text/.. DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -

creativecloudlab

creativecloudlab.com/CclWikipediaCrawler/ text/.. CclWikipediaCrawler/0.3 (url; mail address ) - -

creativecloudlab.com/CclWikipediaCrawler/ text/.. CclWikipediaCrawler/0.1 (url; mail address ) - -

topsy

labs.topsy.com/butterfly/ text/.. Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -

labs.topsy.com/butterfly/ - Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -

gnip

www.gnip.com/ text/.. UnwindFetchor/1.0 (url) - -

www.gnip.com/ image/.. UnwindFetchor/1.0 (url) - -

www.gnip.com/ - UnwindFetchor/1.0 (url) - -

gamedipper

www.gamedipper.com application/json gamedipper.com bot (url) - -

okian

www.okian.ro/ text/.. MyBot/1.0 (url) - -

bsurprised

bsurprised.com/ text/.. BSurprised WikiBox 0.1.3 (url) en -

bsurprised.com/ - BSurprised WikiBox 0.1.3 (url) en

bsurprised.com/ text/.. BSurprised WikiBox 0.1.3 (url) af -

bsurprised.com/ text/.. BSurprised WikiBox 0.1.3 (url) nl -

bsurprised.com/ application/x-www-form-urlencoded BSurprised WikiBox 0.1.3 (url) en

mail

go.mail.ru/help/robots text/.. Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -

go.mail.ru/help/robots text/.. Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) - -

go.mail.ru/help/robots image/.. Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -

go.mail.ru/help/robots - Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) -

trendytrack

trendytrack.com text/.. WordPress/3.5; url - -

enwp

enwp.org/User:SDPatrolBot text/.. SDPatrolBot (url) - -

enwp.org/User:Hellknowz/WikiSharpAPI text/.. WikiSharpAPI/0.3 url (C# .NET) - -

zeebox

www.zeebox.com text/.. Zeebox (url) en-us,en;q=0.5 -

www.zeebox.com application/json Zeebox (url) en-us,en;q=0.5 -

daum

tab.search.daum.net/aboutWebSearch.html text/.. Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -

wikiglass

wikiglass.com text/.. url : mail address - -

FeedBurner

www.FeedBurner.com text/.. FeedBurner/1.0 (url) - -

archive-it

archive-it.org/files/site-owners.html image/.. Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -

archive-it.org/files/site-owners.html - Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -

archive-it.org/files/site-owners.html text/.. Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -

kosmix

www.kosmix.com/html/kosmos.html application/xml Mozilla/5.0(compatible;Kosmos/1.0;url) - -

mediawiki

www.mediawiki.org/ text/.. MediaWiki OAI Harvester 0.2 (url) - -

www.mediawiki.org/ - MediaWiki OAI Harvester 0.2 (url) -

www.mediawiki.org/wiki/Extension:RSS text/.. MediaWikiRSS/0.02 (url) / MediaWiki RSS extension - -

fucinamediale

labs.fucinamediale.com text/.. Mozilla/5.0 (compatible; ExperimentalWikiBot/1.0; url) - -

plos

alm2-dev.plos.org application/json Article Level Metrics - url - -

alm.plos.org application/json PLoS Article Level Metrics - url - -

alm.plos.org - PLoS Article Level Metrics - url -

apercite

www.apercite.fr/robot/index.html image/.. Mozilla/5.0 (compatible; Apercite; url) en-us,en;q=0.5 -

in

www.m-culture.in.th text/.. m-culture.in.th (url) - -

archive

www.archive.org/details/archive.org_bot text/.. Mozilla/5.0 (compatible; archive.org_bot url) - -

www.archive.org/details/archive.org_bot text/.. Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20120911.190842 url) - -

archive.org/details/archive.org_bot image/.. Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -

www.archive.org/details/archive.org_bot image/.. Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -

speaktoit

www.speaktoit.com application/json Speaktoit url - -

paper

support.paper.li/entries/20023257-what-is-paper-li text/.. Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -

tineye

tineye.com/crawler.html application/json TinEye/1.1 (url) - -

tineye.com/crawler.html image/.. TinEye/1.1 (url) - -

tineye.com/crawler.html text/.. TinEye/1.1 (url) - -

xbmc

www.xbmc.org image/.. XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -

www.xbmc.org image/.. XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url) - -

www.xbmc.org image/.. XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url) - -

www.xbmc.org text/.. XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -

zipcode

zipcode.us text/.. Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -

weblio

www.weblio.jp/info/crawler.jsp image/.. Mozilla/5.0 (compatible; Webliobot/0.1; url) - -

www.weblio.jp/ text/.. Mozilla/5.0 (compatible; WeblioBot; url) - -

www.weblio.jp/info/crawler.jsp text/.. Mozilla/5.0 (compatible; Webliobot/0.1; url) - -

www.weblio.jp/ text/.. Mozilla/5.0 (compatible; WeblioBot; url) ja -

emining

emining.jp/ text/.. emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -

emining.jp/ - emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -

federatedmedia

federatedmedia.net text/.. Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -

wikimpress

wikimpress.org/ text/.. Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -

wikimpress.org/ - Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -

seokicks

www.seokicks.de/robot.html text/.. Mozilla/5.0 (compatible; SEOkicks-Robot url) - -

picsearch

www.picsearch.com/bot.html text/.. psbot/0.1 (url) - -

www.picsearch.com/bot.html image/.. psbot/0.1 (url) - -

bibalex

archive.bibalex.org/bot/ text/.. Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -

archive.bibalex.org/bot/ image/.. Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -

fivemost

www.fivemost.com application/json url - -

abonti

www.abonti.com text/.. Mozilla/5.0 (compatible; Abonti/0.91 - url) - -

sistrix

crawler.sistrix.net/ text/.. Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -

drupal

drupal.org/ text/.. Drupal (url) - -

drupal.org/ image/.. Drupal (url) - -

drupal.org/ text/.. User-Agent: Drupal (url) - -

drupal.org/ - Drupal (url) -

rcdtokyo

www.rcdtokyo.com/pc2m/ text/.. Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -

www.rcdtokyo.com/pc2m/ - Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -

github

github.com/pauldix/typhoeus/tree/master text/.. Typhoeus - url - -

easybib

content.easybib.com/autocite/ text/.. EasyBib AutoCite (url) - -

content.easybib.com/autocite/ application/json EasyBib AutoCite (url) - -

tweetmeme

tweetmeme.com/ text/.. Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -

hatena

a.hatena.ne.jp/help text/.. Hatena Antenna/0.5 (url) - -

sf

magpierss.sf.net text/.. MagpieRSS/0.7x (url) en-us,en;q=0.5 -

liferea.sf.net/ text/.. Liferea/1.x.x (Linux; es_ES.UTF-8; url) en-us,en;q=0.5 -

liferea.sf.net/ text/.. Liferea/0.x.x (Linux; en_US.UTF-8; url) en-us,en;q=0.5 -

liferea.sf.net/ - Liferea/1.8.3 (Linux; fr_FR.UTF-8; url) -

yoursite

yoursite.com/botinfo text/.. Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -

example

example.com/ text/.. Wiki 0.0 (url; mail address ) - -

example.com/ text/.. WikiExample 0.1 (url; mail address ) - -

example.com/MyCoolToolPage/ application/json MyCoolTool (url) - -

localhost:8888

localhost:8888 image/.. WordPress/3.5; url - -

localhost:8888 text/.. WordPress/3.5; url - -

openindex

www.openindex.io/en/webmasters/spider.html text/.. Mozilla/5.0 (compatible; OpenindexSpider; url) en-us,en-gb,en;q=0.7,*;q=0.3 -

www.openindex.io/en/webmasters/spider.html text/.. Mozilla/5.0 (compatible; OpenindexSpider; url) - -

proximic

www.proximic.com/info/spider.php text/.. Mozilla/5.0 (compatible; proximic; url) - -

n-grams

www.n-grams.org/icorpusbot.html text/.. iCorpusBot (url) es-es,en-us;q=0.7,en;q=0.3 -

moviecus

www.moviecus.com/botcontactinfo.php application/yaml moviecus bot (url) - -

www.moviecus.com/botcontactinfo.php - moviecus bot (url) -

rockpeaks

www.rockpeaks.com/contact text/.. RockPeaks/0.1 (url) - -

alexa

www.alexa.com/site/help/webmasters text/.. ia_archiver (url; mail address ) - -

vermagerd

www.vermagerd.be/wp text/.. WordPress/3.4.2; url - -

netseer

www.netseer.com/crawler.html text/.. Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address ) - -

sentymetr

sentymetr.pl/bot.html application/json Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -

sentymetr.pl/bot.html text/.. Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -

friendofrenia

friendofrenia.com/ text/.. User-Agent: FriendoFrenia (url) - -

friendofrenia.com/ application/json User-Agent: FriendoFrenia (url) - -

netvibes

www.netvibes.com text/.. Netvibes (url) - -

embed

support.embed.ly/ image/.. Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -

support.embed.ly/ text/.. Mozilla/5.0 (compatible; Embedly/0.2; url) - -

spinn3r

spinn3r.com/robot text/.. Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -

textdigger

textdigger.com text/.. Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -

warebay

www.warebay.com/bot.html text/.. Mozilla/5.0 (compatible; WBSearchBot/1.1; url) en-us,en;q=0.5 -

muso

www.muso.com text/.. Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -

chickyrun

chickyrun.tk/ text/.. ChickyBot/1.1 (url; mail address ) - -

rockmelt

rockmelt.com text/.. RockmeltEmbedService (url) - -

linguee

www.linguee.com/bot text/.. Linguee Bot (url; mail address ) - -

www.linguee.com/bot application/json Linguee Bot (url; mail address ) - -

turnitin

www.turnitin.com/robot/crawlerinfo.html text/.. TurnitinBot/2.1 (url) - -

avantbrowser

www.avantbrowser.com text/.. Advanced Browser (url) en-us,en;q=0.5 -

www.avantbrowser.com text/.. Avant Browser (url) en-us,en;q=0.5 -

simplepie

semrush

www.semrush.com/bot.html text/.. Mozilla/5.0 (compatible; SemrushBot/0.95; url) - -

plagiarismcheck

plagiarismcheck.org application/json WikiCrawl 1.0b (url contact-mail: mail address ) - -

stackoverflow

stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-php text/.. Testing for url - -

meta.stackoverflow.com/q/130398 text/.. Mozilla/5.0 (compatible; stackexchangebot/1.0; url) - -

dasdonkey

www.dasdonkey.com text/.. Mozilla/5.0 (compatible; DonkeyBot/0.1; url) - -

superfeedr

superfeedr.com application/xml Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -

superfeedr.com text/.. Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -

superfeedr.com - Superfeedr bot/2.0 url - Please get in touch if we are polling too hard. - -

sonyericsson

www.sonyericsson.com/UAprof/R800xR301.xml image/.. Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -

msai

www.msai.in/uaprof/micromax/X455.xml image/.. url en,hi -

webceo

online.webceo.com text/.. Mozilla/5.0 (compatible; web-ceo-online-bot/1.0; url) - -

newsgator

www.newsgator.com/ text/.. FeedDemon/2.7 (url; Microsoft Windows XP) en-us,en;q=0.5 -

www.newsgator.com text/.. NewsGatorOnline/2.0 (url; 1 subscribers) en-us,en;q=0.5 -

jetbrains

www.jetbrains.com/omea_reader/ text/.. JetBrains Omea Reader 1.0.x (url) en-us,en;q=0.5 -

www.jetbrains.com/omea_reader/ text/.. JetBrains Omea Reader 2.0 Release Candidate 1 (url) en-us,en;q=0.5 -

medu

medu.pl/bot text/.. Mozilla/5.0 (compatible; Medubot/1.0; url) - -

medu.pl/bot - Mozilla/5.0 (compatible; Medubot/1.0; url) -

site-shot

www.site-shot.com/ image/.. Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -

www.site-shot.com/ text/.. Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 KHTML Site-Shot/2.1 (url) Safari/534.34 en-US,* -

feedshow

www.feedshow.com text/.. FeedshowOnline (url) en-us,en;q=0.5 -

www.feedshow.com text/.. Feedshow/x.0 (url; 1 subscriber) en-us,en;q=0.5 -

seomoz

www.seomoz.org/dp/rogerbot text/.. rogerbot/1.0 (url, mail address ) - -

www.seomoz.org/dp/rogerbot text/.. rogerbot/1.0 (url, mail address ) - -

igrec

www.igrec.ca/projects text/.. Wikitionary Text Parser 0.2 (url) - -

owasp

www.owasp.org/index.php/Category:OWASP_DirBuster_Project text/.. DirBuster-0.12 (url) - -

wikiapiary

wikiapiary.com/wiki/User:Bumble_Bee - Python-urllib/2.7 (WikiApiary; Bumble Bee; url) -

wikiapiary.com/wiki/User:Bumble_Bee application/json Python-urllib/2.7 (WikiApiary; Bumble Bee; url) - -

theworldtopbrands

theworldtopbrands.com text/.. WordPress/3.4.2; url - -

metamagazine

metamagazine.com text/.. WordPress/3.4.2; url - -

metamagazine.com - WordPress/3.4.2; url -

pingdom

www.pingdom.com/ text/.. Pingdom.com_bot_version_1.4_(url) - -

www.pingdom.com text/.. Pingdom.com_bot_version_1.4_(url) - -

localhost

localhost text/.. Mozilla/5.0 (compatible; heritrix/2.0.2 url) - -

localhost/wordpress text/.. WordPress/3.5; url - -

yougorhymes

www.yougorhymes.com/site/rhyme-bot text/.. RhymeBot/0.1 (url) - -

www.yougorhymes.com/site/rhyme-bot - RhymeBot/0.1 (url) -

instapaper

www.instapaper.com/ text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -

js-kit

js-kit.com/ text/.. JS-Kit URL Resolver, url - -

fotopedia

www.fotopedia.com application/json Picor (url) - -

onet

wikimedia

tools.wikimedia.de/~para/GeoCommons/ text/.. url - -

tools.wikimedia.de/~para/GeoCommons/ - url -

backgroundswitcher

www.backgroundswitcher.com/ image/.. John's Background Switcher 4.4 (url) - -

www.backgroundswitcher.com/ image/.. John's Background Switcher 4.6 (url) - -

www.backgroundswitcher.com/ text/.. John's Background Switcher 4.3 (url) - -

nationallibrary

www.nationallibrary.fi/ text/.. Mozilla/5.0 (compatible; heritrix/1.14.0url) - -

www.nationallibrary.fi/ - Mozilla/5.0 (compatible; heritrix/1.14.0url) - -

www.nationallibrary.fi/ image/.. Mozilla/5.0 (compatible; heritrix/1.14.0url) - -

pinterest

pinterest.com/ image/.. Pinterest/0.1 url - -

pinterest.com/ text/.. Pinterest/0.1 url - -

grapeshot

www.grapeshot.co.uk/crawler.php text/.. Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -

Anonymouse

Anonymouse.org/ image/.. url (Unix) - -

Anonymouse.org/ text/.. url (Unix) - -

netarkivet

netarkivet.dk/webcrawler/ text/.. Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -

gsd-software

www.gsd-software.com text/.. GSDCrawler (url) - -

blisshq

www.blisshq.com image/.. bliss/20121128 url - -

www.blisshq.com text/.. bliss/20121128 url - -

wotbox

www.wotbox.com/bot/ text/.. Wotbox/2.01 (url) - -

jetsli

jetsli.de/crawler text/.. Mozilla/5.0 (compatible; Jetslide; url) en-us -

duckduckgo

duckduckgo.com/duckduckbot.html text/.. DuckDuckBot/1.1; (url) - -

duckduckgo.com/duckduckpreview.html text/.. DuckDuckPreview/1.0; (url) - -

duckduckgo.com/duckduckpreview.html - DuckDuckPreview/1.0; (url) - -

searchtechnologies

www.searchtechnologies.com text/.. Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -

tinyurl

tinyurl.com/64t5n text/.. Rome Client (url) Ver: 0.9 en-us,en;q=0.5 -

weborking

weborking.com text/.. Weborking(url) - -

weborking.com application/json Weborking(url) - -

go

kc.nict.go.jp/project1/crawl.html text/.. ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -

kc.nict.go.jp/project1/crawl-ja.html text/.. ICC-Crawler (Mozilla-compatible; mail address ; url) ja -

watchmouse

esciudad

www.esciudad.com/ application/json Esciudad/1.0 (url) - -

femfamilyinternational

femfamilyinternational.org/faith text/.. WordPress/3.5; url - -

nb

www.nb.no/vevfangst image/.. Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -

www.nb.no/vevfangst text/.. Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -

tweetedtimes

tweetedtimes.com text/.. Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -

tweetedtimes.com text/.. TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -

heartrails

capture.heartrails.com/ image/.. Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.17 ja,en-us;q=0.7,en;q=0.3 -

capture.heartrails.com/ text/.. Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.17 ja,en-us;q=0.7,en;q=0.3 -

toshiba

www.toshiba.co.jp/rdc/about/crawl_info_en.htm text/.. TosCrawler/Nutch-1.4 (url; ' mail address dot co dot jp') ja,ja-jp;q=0.7,*;q=0.3 -

compspy

www.compspy.com/spider.html text/.. Mozilla/5.0 (compatible; CompSpyBot/1.0; url) - -

134211.550000016 total

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 104,812,910 page requests (mime type text/html only!) per day are considered crawler requests, out of 605,947,230 external requests, which is 17.3%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	google
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
		code.google.com/p/crawler4j/	text/..	crawler4j (url) - -
		www.google.com/feedfetcher.html	-	FeedFetcher-Google; (url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortografia4) - -
		www.google.com/feedfetcher.html	application/xml	FeedFetcher-Google; (url) - -
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: s~redconceptual) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortopedianew) - -
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien3) - -
		code.google.com/appengine	text/..	www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing) - -
		www.google.com/feedfetcher.html	application/json	Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: rarplayer) - -
		desktop.google.com/	text/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -
		www.google.com/feedfetcher.html	application/xml	Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien4) - -
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw) - -
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
		code.google.com/appengine	text/..	Wiki.java 0.27 AppEngine-Google; (url; appid: wikipediatools) - -
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~kasumiremix) - -
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
		code.google.com/appengine	application/json	Mozilla/5.0 AppEngine-Google; (url; appid: s~redconceptual) - -
		code.google.com/appengine	text/..	Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
		code.google.com/appengine	-	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
		code.google.com/appengine	text/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
		code.google.com/appengine	text/..	Mozilla/5.0 AppEngine-Google; (url; appid: s~birthday-stats) - -
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: d24-img) - -
		www.google.com/bot.html	NONE/wikipedia	- Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/coop/cse/cref	text/..	FeedFetcher-Google-CoOp; (url) - -
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl) - -
		code.google.com/appengine	text/..	route-hacker alpha AppEngine-Google; (url; appid: s~routehackers) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~francetiki) - -
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: usawebproxy0) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d24-img) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: boxapp) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~japantiki) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~app3123ak) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~app-cruxbot) - -
		www.google.com/feedfetcher.html	-	Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: worldwide-propaganda) - -
		code.google.com/appengine	image/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: s~wiki2go-hrd) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: threewiki) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~hr-pulsesubscriber) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~drizzlprox) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~theunblock) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pakgalaxy) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kires-roxy) - -
		code.google.com/appengine	application/json	MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~liquid-helium) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~espanatiki) - -
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: boxapp) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: mehproxy) - -
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
		www.google.com/bot.html	application/ogg	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxyusing121) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: your-zone) - -
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url) ja,en-us;q=0.7,en;q=0.3 -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~proxyseekkety) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~italiatiki) - -
		docs.google.com	text/..	Mozilla/5.0 (compatible; GoogleDocs; script; url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tunisistan) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: python-proxy-server) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: atxproxy) - -
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: azshaderserver) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: my-api) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: toom16-10) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: thetechnolust) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: azshaderserver) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebproxy0) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ridemyhell) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: davrasaurs) - -
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 KHTML Chrome/19.0.1084.52 Safari/536.5 AppEngine-Google; (url; appid: seiyukyouen) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vebproxy) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dkoxyserv) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki3) - -
		www.google.com/feedfetcher.html	image/..	FeedFetcher-Google; (url) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: varlopie) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxy-devakishor) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~stremor-crawler) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vadim-proxy) - -
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxy12345) - -
	facebook
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url) - -
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.1 (url) - -
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.0 (url) - -
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.0 (url) - -
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.1 (url) - -
		developers.facebook.com	image/..	facebookplatform/1.0 (url) - -
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.0 (url) - -
		developers.facebook.com	-	facebookplatform/1.0 (url) - -
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) - -
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url) - -
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url) -
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url) - -
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) en -
		www.bing.com/bingbot.htm	application/vnd.php.serialized	Mozilla/5.0 (compatible; bingbot/2.0; url) - -
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5 - -
		www.bing.com/bingbot.htm	application/json	Mozilla/5.0 (compatible; bingbot/2.0; url) - -
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) -
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url) en -
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3 - -
		www.bing.com/bingbot.htm	application/xml	Mozilla/5.0 (compatible; bingbot/2.0; url) - -
		www.bing.com/bingbot.htm	application/ogg	Mozilla/5.0 (compatible; bingbot/2.0; url) - -
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8) - -
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4) - -
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url) -
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url) - -
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url) en-us,en;q=0.5 -
		www.google.com/bot.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/bot.html	application/json	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
		www.google.com/bot.html	-	GoogleBot/2.1 (url) en-us,en;q=0.5
		www.google.com/bot.html	application/xml	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRJ/YATS crawler (url) - -
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	image/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp; url) en-us,en;q=0.5 -
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
		developer.yahoo.com/yql/provider	text/..	Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRT/1.0 crawler (url) - -
		help.yahoo.com/help/us/ysearch/slurp	application/xml	Mozilla/5.0 (compatible; Yahoo! Slurp;url) - -
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRW/1.0 crawler (url) ja, *;q=0.5 -
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRW/1.0 crawler (url) - -
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url) -
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
		yandex.com/bots	application/ogg	Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
		yandex.com/bots	application/json	Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
		www.baidu.com/search/spider.htm	image/..	Baiduspider-image(url) - -
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
		www.baidu.com/search/spider.htm	text/..	Baiduspider-image(url) - -
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
		www.baidu.com/search/spider.html	application/json	Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
		www.baidu.com/search/spider.html	application/json	Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
		www.baidu.com/search/spider.html	application/xml	Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
	msn
		search.msn.com/msnbot.htm	text/..	msnbot-NewsBlogs/2.0b (url) - -
		search.msn.com/msnbot.htm	text/..	msnbot-media/1.1 (url) - -
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url) - -
		search.msn.com/msnbot.htm	image/..	msnbot-media/1.1 (url) - -
		search.msn.com/msnbot.htm	image/..	msnbot-NewsBlogs/2.0b (url) - -
		search.msn.com/msnbot.htm	text/..	msnbot-Products/1.0 (url) - -
		search.msn.com/msnbot.htm	image/..	msnbot/2.0b (url) - -
		search.msn.com/msnbot.htm	-	msnbot-media/1.1 (url) - -
		search.msn.com/msnbot.htm	text/..	msnbot-UDiscovery/2.0b (url) - -
		search.msn.com/msnbot.htm	text/..	msnbot/0.01 (url) - -
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) - -
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
		help.naver.com/robots/	image/..	Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
		help.naver.com/robots/	text/..	Yeti/1.1 (NHN Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) tr,en;q=0.5 -
		help.naver.com/robots/	application/json	Yeti/1.0 (NHN Corp.; url) - -
	cibra
		cibra.de/	text/..	CiBra Data Collector (url) - -
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
		www.youdao.com/help/reader/faq/topic006/	-	Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;url;1 subscribers;) - -
	genieo
		www.genieo.com/webfilter.html	text/..	Mozilla/5.0 (compatible; Genieo/1.0 url) - -
		www.genieo.com/webfilter.html	application/xml	Mozilla/5.0 (compatible; Genieo/1.0 url) - -
		www.genieo.com/webfilter.html	image/..	Mozilla/5.0 (compatible; Genieo/1.0 url) - -
		www.genieo.com/webfilter.html	image/..	Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
	sblog
		fulltext.sblog.cz/screenshot/	image/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url) cs -
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
		fulltext.sblog.cz/	-	SeznamBot/3.0 (url) cs -
		fulltext.sblog.cz/screenshot/	-	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
		fulltext.sblog.cz/screenshot/	-	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) - -
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
	soso
		help.soso.com/webspider.htm	text/..	Mozilla/5.0(compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
		help.soso.com/webspider.htm	text/..	Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
		help.soso.com/webspider.htm	-	Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
		help.soso.com/webspider.htm	image/..	Sosospider(url) zh-cn,zh-hk,zh-tw,en-us -
		help.soso.com/webspider.htm	text/..	Mozilla/5.0(compatible; Sosospider/2.0; url) - -
		help.soso.com/webspider.htm	text/..	Sosospider(url) - -
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
		ahrefs.com/robot/	-	Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
		ahrefs.com/robot/	-	Mozilla/5.0 (compatible; AhrefsBot/4.0; url) -
		ahrefs.com/robot/	application/json	Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url) - -
		help.zum.com/inquiry	image/..	ZumBot/1.0 (ZUM Search; url) - -
	blekko
		blekko.com/about/blekkobot	text/..	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
		blekko.com/about/blekkobot	-	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
	php
		pear.php.net/	application/vnd.php.serialized	PEAR HTTP_Request class ( url ) - -
		pear.php.net/package/http_request2	text/..	HTTP_Request2/0.5.2 (url) PHP/5.2.17 - -
		pear.php.net/	image/..	PEAR HTTP_Request class ( url ) - -
		pear.php.net/	text/..	PEAR HTTP_Request class ( url ) - -
		pear.php.net/package/http_request2	application/xml	HTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
		pear.php.net/	application/xml	PEAR HTTP_Request class ( url ) - -
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.17 - -
		pear.php.net/package/http_request2	image/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15 - -
		pear.php.net/package/http_request2	image/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url) - -
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
		www.exabot.com/go/robot	-	Mozilla/5.0 (compatible; Exabot/3.0; url) - -
		www.exabot.com/go/robot	-	Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
	143
		173.13.143.74/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot; url) - -
		173.13.143.74/bot.php	-	Mozilla/5.0 (compatible; YioopBot; url) - -
		173.13.143.74/bot.php	image/..	Mozilla/5.0 (compatible; YioopBot; url) - -
	echonest
		the.echonest.com/reader/	application/xml	nestReader/0.3 (discovery; url; reader at echonest.com) en -
		the.echonest.com/reader/	text/..	nestReader/0.3 (discovery; url; reader at echonest.com) en -
	toolserver
		wiki.toolserver.org/view/GeoHack	text/..	Geohack (url) - -
		wiki.toolserver.org/view/GeoHack	-	Geohack (url) -
		toolserver.org/~dispenser/	-	CacheThumbs/1.2 (url) -
		toolserver.org/~dispenser/	image/..	CacheThumbs/1.2 (url) - -
		toolserver.org/~dispenser/	text/..	CacheThumbs/1.2 (url) - -
		toolserver.org/~dispenser/	text/..	DispensersTools (url) - -
		toolserver.org/~dispenser/	-	DispensersTools (url) -
		toolserver.org/~dispenser/	application/x-www-form-urlencoded	DispensersTools (url) -
		toolserver.org/~para/cgi-bin/kmlexport	text/..	url libwww-perl/6.02 - -
		toolserver.org/~dispenser/	application/json	DispensersTools (url) - -
	wordpress
		josefboberg.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
		tsjok45.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
		kabarislam.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
		greatriversofhope.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
		jaggedykaye.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
		lesliebrodie.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
		beyondthelies.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
		gunnyg.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
		ambiya999dotcom.wordpress.com	text/..	WordPress/3.6-alpha-23288; url - -
	www.
		www.	text/..	GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
		www.	text/..	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
		www.	text/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.20.0 url - -
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.0 url - -
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WPCleaner (url) - -
		en.wikipedia.org/wiki/Wikipedia:Huggle	-	Huggle/2.1.19.0 url -
		fr.wikipedia.org/wiki/Utilisateur:Salebot	application/json	Salebot, see url (uses Perl MediaWiki::API) - -
		sk.wikipedia.org/wiki/Redaktor:TeslaBot	-	TeslaBot (url) - -
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url) zh-cn -
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url) zh-cn -
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url) - -
		www.sogou.com/docs/help/webmasters.htm#07	application/json	Sogou web spider/4.0(url) - -
		www.sogou.com/docs/help/webmasters.htm#07	image/..	Sogou Pic Spider/3.0(url) zh-cn -
	finecomb
		finecomb.com/	-	api/1.1 (url; mail address ) - -
		finecomb.com/	application/json	api/1.1 (url; mail address ) - -
	wikidict
		www.wikidict.de	text/..	url - -
	wwwgogetpapers
		wwwgogetpapers.com/	application/json	User-Agent: GoGetPapersBot (url) - -
	traslated
		mymemory.traslated.net/doc/	text/..	Mozilla/5.0 (MyMemory Bot url) - -
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
		www.majestic12.co.uk/bot.php?	-	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) -
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url) - -
		shoulu.jike.com/spider.html	-	Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
		shoulu.jike.com/spider.html	image/..	Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
	zookabot
		zookabot.com	text/..	Zookabot/2.5;url - -
		zookabot.com	image/..	Zookabot/2.5;url - -
	coccoc
		help.coccoc.vn/	text/..	coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
		help.coccoc.vn/	-	coccoc/1.0 (url) en-us;q=0.7,en;q=0.3 -
		help.coccoc.vn/	text/..	coccoc/1.0 (url) - -
		help.coccoc.vn/	text/..	coccoc/1.0 (url) en-us,en;q=0.5 -
	flipboard
		flipboard.com/browserproxy	image/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
		flipboard.com/browserproxy	application/json	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url) en-us,en;q=0.5 -
		flipboard.com/browserproxy	image/..	null (FlipboardProxy/1.1; url) - -
		flipboard.com/browserproxy	-	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
		flipboard.com/browserproxy	-	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
	yacy
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 FreeBSD 9.1-RELEASE; java 1.7.0_011; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.6.11-1-ARCH; java 1.7.0_03; Europe/fr) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.6.11-1-ARCH; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_24; Europe/fr) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; x86_64 Mac OS X 10.8.1; java 1.6.0_37; Asia/ru) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.7.4-1-ARCH; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_07; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (torworld/any; amd64 Linux 3.4.0-5-generic; java 1.7.0_09; Europe/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (webportal/global; x86 Windows 7 6.1; java 1.6.0_18; Europe/fr) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (webportal-global; x86 Windows XP 5.1; java 1.7.0_07; Europe/de) url en-us,en;q=0.5 -
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.5.0-21-generic; java 1.7.0_09; America/en) url en-us,en;q=0.5 -
		yacy.net/bot.html	-	yacybot (torworld/any; amd64 Linux 3.4.0-5-generic; java 1.7.0_09; Europe/en) url en-us,en;q=0.5 -
	SearchNearMe
		SearchNearMe.com/contact.php	application/vnd.php.serialized	SearchNearMe (url) - -
		SearchNearMe.com/contact.php	text/..	SearchNearMe (url) - -
	elcidharth
		elcidharth.com	text/..	WordPress/3.6-alpha-23288; url - -
		elcidharth.com	image/..	WordPress/3.6-alpha-23288; url - -
	ac
		www.tkl.iis.u-tokyo.ac.jp/~crawler/	text/..	Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
		www.clips.ua.ac.be/pages/pattern	application/json	Pattern/2.3 url - -
		www.tkl.iis.u-tokyo.ac.jp/~crawler/	-	Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
		www.ninjal.ac.jp/corpus_center/ulc/crawl-en	text/..	Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
		www.tkl.iis.u-tokyo.ac.jp/~crawler/	text/..	Mozilla/5.0 (compatible; Steeler/3.5; url) - -
		www.tkl.iis.u-tokyo.ac.jp/~crawler/	image/..	Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
	cognarius
		cognarius.com	-	AppsArlak/1.0 (url) -
		cognarius.com	application/json	AppsArlak/1.0 (url) - -
		cognarius.com	text/..	AppsArlak/1.0 (url) - -
	bin-co
		www.bin-co.com/php/scripts/load/	text/..	BinGet/1.00.A (url) - -
		www.bin-co.com/php/scripts/load/	application/vnd.php.serialized	BinGet/1.00.A (url) - -
	goo
		help.goo.ne.jp/contact/	text/..	goo wikipedia (url) - -
		goo.gl/7y4SX	text/..	GoogleProducer; (url) - -
		search.goo.ne.jp/option/use/sub4/sub4-1/	text/..	ichiro/3.0 (url) - -
		help.goo.ne.jp/door/crawler.html	text/..	ichiro/3.0 (url) - -
		search.goo.ne.jp/option/use/sub4/sub4-1/	-	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
		search.goo.ne.jp/option/use/sub4/sub4-1/	text/..	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
		goo.gl/7y4SX	image/..	GoogleProducer; (url) - -
		search.goo.ne.jp/option/use/sub4/sub4-1/	-	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
		search.goo.ne.jp/option/use/sub4/sub4-1/	text/..	DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
	creativecloudlab
		creativecloudlab.com/CclWikipediaCrawler/	text/..	CclWikipediaCrawler/0.3 (url; mail address ) - -
		creativecloudlab.com/CclWikipediaCrawler/	text/..	CclWikipediaCrawler/0.1 (url; mail address ) - -
	topsy
		labs.topsy.com/butterfly/	text/..	Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
		labs.topsy.com/butterfly/	-	Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url) - -
		www.gnip.com/	image/..	UnwindFetchor/1.0 (url) - -
		www.gnip.com/	-	UnwindFetchor/1.0 (url) - -
	gamedipper
		www.gamedipper.com	application/json	gamedipper.com bot (url) - -
	okian
		www.okian.ro/	text/..	MyBot/1.0 (url) - -
	bsurprised
		bsurprised.com/	text/..	BSurprised WikiBox 0.1.3 (url) en -
		bsurprised.com/	-	BSurprised WikiBox 0.1.3 (url) en
		bsurprised.com/	text/..	BSurprised WikiBox 0.1.3 (url) af -
		bsurprised.com/	text/..	BSurprised WikiBox 0.1.3 (url) nl -
		bsurprised.com/	application/x-www-form-urlencoded	BSurprised WikiBox 0.1.3 (url) en
	mail
		go.mail.ru/help/robots	text/..	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
		go.mail.ru/help/robots	text/..	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) - -
		go.mail.ru/help/robots	image/..	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
		go.mail.ru/help/robots	-	Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; url) -
	trendytrack
		trendytrack.com	text/..	WordPress/3.5; url - -
	enwp
		enwp.org/User:SDPatrolBot	text/..	SDPatrolBot (url) - -
		enwp.org/User:Hellknowz/WikiSharpAPI	text/..	WikiSharpAPI/0.3 url (C# .NET) - -
	zeebox
		www.zeebox.com	text/..	Zeebox (url) en-us,en;q=0.5 -
		www.zeebox.com	application/json	Zeebox (url) en-us,en;q=0.5 -
	daum
		tab.search.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
	wikiglass
		wikiglass.com	text/..	url : mail address - -
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url) - -
	archive-it
		archive-it.org/files/site-owners.html	image/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
		archive-it.org/files/site-owners.html	-	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
		archive-it.org/files/site-owners.html	text/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) - -
	kosmix
		www.kosmix.com/html/kosmos.html	application/xml	Mozilla/5.0(compatible;Kosmos/1.0;url) - -
	mediawiki
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url) - -
		www.mediawiki.org/	-	MediaWiki OAI Harvester 0.2 (url) -
		www.mediawiki.org/wiki/Extension:RSS	text/..	MediaWikiRSS/0.02 (url) / MediaWiki RSS extension - -
	fucinamediale
		labs.fucinamediale.com	text/..	Mozilla/5.0 (compatible; ExperimentalWikiBot/1.0; url) - -
	plos
		alm2-dev.plos.org	application/json	Article Level Metrics - url - -
		alm.plos.org	application/json	PLoS Article Level Metrics - url - -
		alm.plos.org	-	PLoS Article Level Metrics - url -
	apercite
		www.apercite.fr/robot/index.html	image/..	Mozilla/5.0 (compatible; Apercite; url) en-us,en;q=0.5 -
	in
		www.m-culture.in.th	text/..	m-culture.in.th (url) - -
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url) - -
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20120911.190842 url) - -
		archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
		www.archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
	speaktoit
		www.speaktoit.com	application/json	Speaktoit url - -
	paper
		support.paper.li/entries/20023257-what-is-paper-li	text/..	Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
	tineye
		tineye.com/crawler.html	application/json	TinEye/1.1 (url) - -
		tineye.com/crawler.html	image/..	TinEye/1.1 (url) - -
		tineye.com/crawler.html	text/..	TinEye/1.1 (url) - -
	xbmc
		www.xbmc.org	image/..	XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url) - -
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url) - -
		www.xbmc.org	text/..	XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url) - -
	zipcode
		zipcode.us	text/..	Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
	weblio
		www.weblio.jp/info/crawler.jsp	image/..	Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
		www.weblio.jp/	text/..	Mozilla/5.0 (compatible; WeblioBot; url) - -
		www.weblio.jp/info/crawler.jsp	text/..	Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
		www.weblio.jp/	text/..	Mozilla/5.0 (compatible; WeblioBot; url) ja -
	emining
		emining.jp/	text/..	emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
		emining.jp/	-	emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
	federatedmedia
		federatedmedia.net	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
	wikimpress
		wikimpress.org/	text/..	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
		wikimpress.org/	-	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0 - -
	seokicks
		www.seokicks.de/robot.html	text/..	Mozilla/5.0 (compatible; SEOkicks-Robot url) - -
	picsearch

Page requests for probable crawlers, recognized by keyword

Count
x 1000 Agent string

Mime type (count ≥ 3)

PythonWikipediaBot/1.0 - -

application/json

application/xml

text/..

-

application/x-www-form-urlencoded

image/..

application/pdf

PythonWikipediaBot/1.0 -

-

application/x-www-form-urlencoded

spider - -

text/..

application/vnd.php.serialized

application/yaml

application/json

image/..

-

AniBot/0.9 php/curl - -

application/vnd.php.serialized

-

text/..

image/..

php wikibot classes - -

application/vnd.php.serialized

text/..

-

MediaWikiCrawler-Google/2.0 ( mail address ) - -

text/..

-

GoogleBot-Image/1.0 - -

image/..

text/..

-

LinkParser/2.0 - -

text/..

DotNetWikiBot/2.101 (Unix 2.6.32.39; ) - -

text/..

application/xml

Peachy MediaWiki Bot API Version 1.0 - -

application/vnd.php.serialized

text/..

DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -

text/..

application/xml

Peachy MediaWiki Bot API Version 1.0 -

-

wikiwix-bot-3.0 - -

text/..

-

tigerbot - -

application/json

text/..

-

Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -

text/..

-

application/pdf

Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -

text/..

GoogleBot-Image/1.0 - -

text/..

image/..

-

application/rsd+xml

ClueBot/1.1 - -

application/vnd.php.serialized

text/..

Answersbot - -

text/..

DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -

text/..

application/xml

Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider - -

text/..

application/xml

application/json

application/vnd.php.serialized

ClueBot/2.0 - -

application/vnd.php.serialized

www.integromedb.org/Crawler - -

text/..

-

application/xml

TrueKnowledgeBot bot mail address > - -

application/xml

application/vnd.php.serialized

image/..

Pywikipediabot/2.0 - -

application/json

text/..

gsa-crawler (Enterprise; T3-P9JWVCTT9WWGY; mail address ) - -

text/..

Pywikipediabot/2.0 -

application/x-www-form-urlencoded

Wikipath Bot (email: mail address ) - -

application/json

Mozilla 5.0 (Apibot 0.32) - -

application/vnd.php.serialized

text/..

crawl/0.4 libcrawl/0.3 - -

text/..

image/..

application/json

application/ogg

DigitalsmithsBot - -

text/..

Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -

text/..

application/json

-

image/..

Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider zh-CN -

text/..

-

image/..

application/ogg

Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -

image/..

text/..

application/json

CorenSearchBot/1.7 en libwww-perl/6.04 - -

text/..

unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*

-

DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -

text/..

application/x-www-form-urlencoded

MediaWiki::Bot/3.2.6 - -

application/json

-

text/..

DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -

text/..

application/xml

WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -

text/..

application/json

AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -

application/json

AarghBot Linux - -

text/..

Tawbot (public svn release; plwiki) - -

text/..

WikiCatResearchBot ( mail address ) - -

text/..

Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-us -

image/..

application/json

text/..

-

mail address - -

application/vnd.php.serialized

text/..

HTMLParser/1.6 - -

text/..

-

mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -

application/json

application/x-www-form-urlencoded

DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -

text/..

application/xml

image/..

application/ogg

www.monit24.pl-m24Bot/4.0- - -

-

image/..

text/..

Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -

text/..

ExactusBot-v0.1 - -

text/..

DotNetWikiBot/2.100 (Unix 3.2.0.35; ) - -

text/..

Hulubot - -

text/..

gsa-crawler (Enterprise; T3-F5C5JE7XKWWBK; mail address ) - -

text/..

-

Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -

text/..

-

SchoolReviewNetworkWikiBot - -

application/json

text/..

XBot v1.0 using MER-C's Wiki.java - -

text/..

AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -

application/json

SearchBot - -

text/..

SineBot/1.5.19(User:SineBot) - -

application/vnd.php.serialized

text/..

dtSearchSpider - -

text/..

php wikibot classes -

-

DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -

text/..

application/xml

YBot/0.1 - -

application/vnd.php.serialized

GermCrawler - -

application/json

text/..

AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -

application/json

Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1 - -

image/..

text/..

-

Test Webbot - -

text/..

application/json

MyCuteBot/0.1 - -

text/..

application/json

application/vnd.php.serialized

DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; ) - -

text/..

application/xml

- Peachy MediaWiki Bot API Version 1.0 -

multipart/form-data

mySpider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 -

text/..

-

Phantom.js bot cs-CZ,en,* -

image/..

text/..

-

AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -

application/json

MediaWiki::Bot/3.005002 - -

application/json

360spider-image - -

image/..

text/..

milog_bot/1.0 ( mail address ) - -

text/..

HosiryuhosiBot IRC-RecentChanges Checker ja -

text/..

application/x-www-form-urlencoded

Spinuf Spider - -

text/..

-

Twitterbot/1.0 - -

text/..

image/..

-

application/pdf

JavaCrawler/1.1 - -

text/..

SiocWikiBot/1.0 - -

application/vnd.php.serialized

text/..

SurakWare MediaWiki Bot/1.0 - -

text/..

DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -

text/..

application/xml

AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -

application/json

text/..

- PythonWikipediaBot/1.0 -

multipart/form-data

Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -

application/xml

text/..

crawler - -

image/..

text/..

wikiwix-bot-3.0 en -

text/..

AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -

application/json

text/..

GoogleBot-Image/1.0 -

-

image/..

OrlodrimBot/1.0 - -

text/..

-

application/x-www-form-urlencoded

HRoestBot, de-wikipedia using pywikipedia framework -

application/x-www-form-urlencoded

-

WPBot 1.0 - -

text/..

image/..

Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -

image/..

text/..

~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) - -

text/..

XLinkBot/1.00 - -

text/..

HRoestBot, de-wikipedia using pywikipedia framework - -

text/..

application/json

Junut Bot 1.0.3 ru-RU,en,* -

text/..

-

application/xml

~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) -

-

Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl - -

text/..

-

HTMLParser/2.0 - -

text/..

-

application/xml

WikiBot/0.1 - -

text/..

image/..

TicketsBot/0.1 - -

text/..

FAST Search Web Crawler 14.0.0325.0000 -

-

text/..

FAST Search Web Crawler 14.0.0325.0000 - -

text/..

-

image/..

application/ogg

DotNetWikiBot/2.101 (Unix 2.6.32.45; ) - -

text/..

DotNetWikiBot/2.101 (Unix 3.0.0.12; ) - -

text/..

application/xml

COIBot/1.00 - -

text/..

MBBot/1.0.0 en-us,en;q=0.5 -

text/..

Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -

text/..

image/..

Zing-BottaBot/2.0 - -

text/..

TVersity Media Robot - -

text/..

Empedia Bot - -

text/..

wikbotlite/2.0 CFNetwork/609 Darwin/13.0.0 en-us -

image/..

application/json

text/..

UCMore Crawler App en-us,en;q=0.5 -

text/..

Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot en-us,en;q=0.5 -

text/..

theWxitBot/0.1 - -

application/json

image/..

Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -

text/..

Metabot 0.1 - -

text/..

Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 416 subscribers; feed-id=3852576738117026533) - -

application/xml

-

DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -

text/..

HersfoldIRCBot version 1.2.1 - -

text/..

-

Geni ircpybot 1.0 - -

application/json

text/..

application/xml

Goalkeeperbot(User:Beetstra)/1.0 - -

text/..

COIBot/2.0 - -

text/..

unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8

-

Wikibot/2.0.2 CFNetwork/548.1.4 Darwin/11.0.0 en-us -

image/..

application/json

text/..

Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -

text/..

image/..

Webwiki Search Engine Bot - www.webwiki.de - -

text/..

python-wikitools/1.2 (User:BernsteinBot) -

application/x-www-form-urlencoded

Mozilla/5.0 (compatible; NewThingForMeBot; mailto: mail address ) - -

text/..

image/..

Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -

text/..

-

python-wikitools/1.2 (User:BernsteinBot) - -

application/json

DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -

text/..

application/xml

Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1 - -

image/..

text/..

-

IsraBot - -

text/..

MerlBot -

-

AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -

application/json

Geni ircpybot 1.0 -

-

LWNutch/Nutch-1.4 (another scientific bot - we accept your robots.txt! ) en-us,en-gb,en;q=0.7,*;q=0.3 -

text/..

Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-gb -

image/..

application/json

text/..

-

LauschenBot/1.1 ( mail address ) - -

text/..

python-wikitools/1.2 (User:LaraBot) - -

application/json

LauschenBot/1.1 ( mail address ) -

-

Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 de-de -

image/..

application/json

text/..

-

FAST Enterprise Crawler 6 used by contosoa.com ( mail address ) - -

text/..

MerlBot - -

application/vnd.php.serialized

AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -

application/json

python-wikitools/1.2 (User:LaraBot) -

application/x-www-form-urlencoded

Mozilla 5.0 (Apibot 0.30b5) - -

application/vnd.php.serialized

DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -

text/..

application/xml

IsraBot -

-

NoyaBot - -

text/..

Wikibot/2.0.1 CFNetwork/609 Darwin/13.0.0 en-us -

image/..

application/json

text/..

Robots.txt finder - -

text/..

GoogleBot - -

text/..

Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 zh-cn -

image/..

application/json

text/..

Local Site Parser 1.0 en-us,en;q=0.5 -

text/..

Mozilla/5.0 (Bgbot 0.5) - -

text/..

OpenLink Virtuoso RDF crawler - -

image/..

text/..

-

Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.2/1.8.5.168;) Opera Mini/3.1 - -

image/..

text/..

-

bitlybot - -

text/..

image/..

-

MediaWiki::PerlBot/User:Wdwdbot - -

application/json

bot: fr-anal - -

application/json

MediaWiki::Bot 3.1.5 -

application/x-www-form-urlencoded

AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -

application/json

MediaWiki::Bot 3.1.5 - -

application/json

SoxBot PHP -

-

Inlibris.com XMLBot/1.0 - -

text/..

EarwigBot/0.2.dev.git4ff7612a (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) -

application/x-www-form-urlencoded

DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -

text/..

application/xml

39687.77 total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

Generated on Tue, Dec 24, 2013 13:08
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	PythonWikipediaBot/1.0 - -
		application/json
		application/xml
		text/..
		-
		application/x-www-form-urlencoded
		image/..
		application/pdf
	PythonWikipediaBot/1.0 -
		-
		application/x-www-form-urlencoded
	spider - -
		text/..
		application/vnd.php.serialized
		application/yaml
		application/json
		image/..
		-
	AniBot/0.9 php/curl - -
		application/vnd.php.serialized
		-
		text/..
		image/..
	php wikibot classes - -
		application/vnd.php.serialized
		text/..
		-
	MediaWikiCrawler-Google/2.0 ( mail address ) - -
		text/..
		-
	GoogleBot-Image/1.0 - -
		image/..
		text/..
		-
	LinkParser/2.0 - -
		text/..
	DotNetWikiBot/2.101 (Unix 2.6.32.39; ) - -
		text/..
		application/xml
	Peachy MediaWiki Bot API Version 1.0 - -
		application/vnd.php.serialized
		text/..
	DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
		text/..
		application/xml
	Peachy MediaWiki Bot API Version 1.0 -
		-
	wikiwix-bot-3.0 - -
		text/..
		-
	tigerbot - -
		application/json
		text/..
		-
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
		text/..
		-
		application/pdf
	Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
		text/..
	GoogleBot-Image/1.0 - -
		text/..
		image/..
		-
		application/rsd+xml
	ClueBot/1.1 - -
		application/vnd.php.serialized
		text/..
	Answersbot - -
		text/..
	DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
		text/..
		application/xml
	Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider - -
		text/..
		application/xml
		application/json
		application/vnd.php.serialized
	ClueBot/2.0 - -
		application/vnd.php.serialized
	www.integromedb.org/Crawler - -
		text/..
		-
		application/xml
	TrueKnowledgeBot bot mail address > - -
		application/xml
		application/vnd.php.serialized
		image/..
	Pywikipediabot/2.0 - -
		application/json
		text/..
	gsa-crawler (Enterprise; T3-P9JWVCTT9WWGY; mail address ) - -
		text/..
	Pywikipediabot/2.0 -
		application/x-www-form-urlencoded
	Wikipath Bot (email: mail address ) - -
		application/json
	Mozilla 5.0 (Apibot 0.32) - -
		application/vnd.php.serialized
		text/..
	crawl/0.4 libcrawl/0.3 - -
		text/..
		image/..
		application/json
		application/ogg
	DigitalsmithsBot - -
		text/..
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
		text/..
		application/json
		-
		image/..
	Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider zh-CN -
		text/..
		-
		image/..
		application/ogg
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
		image/..
		text/..
		application/json
	CorenSearchBot/1.7 en libwww-perl/6.04 - -
		text/..
	unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
		-
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
		text/..
		application/x-www-form-urlencoded
	MediaWiki::Bot/3.2.6 - -
		application/json
		-
		text/..
	DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
		text/..
		application/xml
	WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
		text/..
		application/json
	AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
		application/json
	AarghBot Linux - -
		text/..
	Tawbot (public svn release; plwiki) - -
		text/..
	WikiCatResearchBot ( mail address ) - -
		text/..
	Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-us -
		image/..
		application/json
		text/..
		-
	mail address - -
		application/vnd.php.serialized
		text/..
	HTMLParser/1.6 - -
		text/..
		-
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
		application/json
		application/x-www-form-urlencoded
	DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
		text/..
		application/xml
		image/..
		application/ogg
	www.monit24.pl-m24Bot/4.0- - -
		-
		image/..
		text/..
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
		text/..
	ExactusBot-v0.1 - -
		text/..
	DotNetWikiBot/2.100 (Unix 3.2.0.35; ) - -
		text/..
	Hulubot - -
		text/..
	gsa-crawler (Enterprise; T3-F5C5JE7XKWWBK; mail address ) - -
		text/..
		-
	Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
		text/..
		-
	SchoolReviewNetworkWikiBot - -
		application/json
		text/..
	XBot v1.0 using MER-C's Wiki.java - -
		text/..
	AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
		application/json
	SearchBot - -
		text/..
	SineBot/1.5.19(User:SineBot) - -
		application/vnd.php.serialized
		text/..
	dtSearchSpider - -
		text/..
	php wikibot classes -
		-
	DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
		text/..
		application/xml
	YBot/0.1 - -
		application/vnd.php.serialized
	GermCrawler - -
		application/json
		text/..
	AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
		application/json
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1 - -
		image/..
		text/..
		-
	Test Webbot - -
		text/..
		application/json
	MyCuteBot/0.1 - -
		text/..
		application/json
		application/vnd.php.serialized
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; ) - -
		text/..
		application/xml
	- Peachy MediaWiki Bot API Version 1.0 -
		multipart/form-data
	mySpider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 -
		text/..
		-
	Phantom.js bot cs-CZ,en,* -
		image/..
		text/..
		-
	AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
		application/json
	MediaWiki::Bot/3.005002 - -
		application/json
	360spider-image - -
		image/..
		text/..
	milog_bot/1.0 ( mail address ) - -
		text/..
	HosiryuhosiBot IRC-RecentChanges Checker ja -
		text/..
		application/x-www-form-urlencoded
	Spinuf Spider - -
		text/..
		-
	Twitterbot/1.0 - -
		text/..
		image/..
		-
		application/pdf
	JavaCrawler/1.1 - -
		text/..
	SiocWikiBot/1.0 - -
		application/vnd.php.serialized
		text/..
	SurakWare MediaWiki Bot/1.0 - -
		text/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -
		text/..
		application/xml
	AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
		application/json
		text/..
	- PythonWikipediaBot/1.0 -
		multipart/form-data
	Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
		application/xml
		text/..
	crawler - -
		image/..
		text/..
	wikiwix-bot-3.0 en -
		text/..
	AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
		application/json
		text/..
	GoogleBot-Image/1.0 -
		-
		image/..
	OrlodrimBot/1.0 - -
		text/..
		-
		application/x-www-form-urlencoded
	HRoestBot, de-wikipedia using pywikipedia framework -
		application/x-www-form-urlencoded
		-
	WPBot 1.0 - -
		text/..
		image/..
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
		image/..
		text/..
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) - -
		text/..
	XLinkBot/1.00 - -
		text/..
	HRoestBot, de-wikipedia using pywikipedia framework - -
		text/..
		application/json
	Junut Bot 1.0.3 ru-RU,en,* -
		text/..
		-
		application/xml
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address ) -
		-
	Mozilla/5.0 (compatible; Mail.RU/3.14) CrawlMl - -
		text/..
		-
	HTMLParser/2.0 - -
		text/..
		-
		application/xml
	WikiBot/0.1 - -
		text/..
		image/..
	TicketsBot/0.1 - -
		text/..
	FAST Search Web Crawler 14.0.0325.0000 -
		-
		text/..
	FAST Search Web Crawler 14.0.0325.0000 - -
		text/..
		-
		image/..
		application/ogg
	DotNetWikiBot/2.101 (Unix 2.6.32.45; ) - -
		text/..
	DotNetWikiBot/2.101 (Unix 3.0.0.12; ) - -
		text/..
		application/xml
	COIBot/1.00 - -
		text/..
	MBBot/1.0.0 en-us,en;q=0.5 -
		text/..
	Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
		text/..
		image/..
	Zing-BottaBot/2.0 - -
		text/..
	TVersity Media Robot - -
		text/..
	Empedia Bot - -
		text/..
	wikbotlite/2.0 CFNetwork/609 Darwin/13.0.0 en-us -
		image/..
		application/json
		text/..
	UCMore Crawler App en-us,en;q=0.5 -
		text/..
	Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot en-us,en;q=0.5 -
		text/..
	theWxitBot/0.1 - -
		application/json
		image/..
	Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
		text/..
	Metabot 0.1 - -
		text/..
	Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 416 subscribers; feed-id=3852576738117026533) - -
		application/xml
		-
	DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
		text/..
	HersfoldIRCBot version 1.2.1 - -
		text/..
		-
	Geni ircpybot 1.0 - -
		application/json
		text/..
		application/xml
	Goalkeeperbot(User:Beetstra)/1.0 - -
		text/..
	COIBot/2.0 - -
		text/..
	unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
		-
	Wikibot/2.0.2 CFNetwork/548.1.4 Darwin/11.0.0 en-us -
		image/..
		application/json
		text/..
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
		text/..
		image/..
	Webwiki Search Engine Bot - www.webwiki.de - -
		text/..
	python-wikitools/1.2 (User:BernsteinBot) -
		application/x-www-form-urlencoded
	Mozilla/5.0 (compatible; NewThingForMeBot; mailto: mail address ) - -
		text/..
		image/..
	Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
		text/..
		-
	python-wikitools/1.2 (User:BernsteinBot) - -
		application/json
	DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
		text/..
		application/xml
	Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1 - -
		image/..
		text/..
		-
	IsraBot - -
		text/..
	MerlBot -
		-
	AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
		application/json
	Geni ircpybot 1.0 -
		-
	LWNutch/Nutch-1.4 (another scientific bot - we accept your robots.txt! ) en-us,en-gb,en;q=0.7,*;q=0.3 -
		text/..
	Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 en-gb -
		image/..
		application/json
		text/..
		-
	LauschenBot/1.1 ( mail address ) - -
		text/..
	python-wikitools/1.2 (User:LaraBot) - -
		application/json
	LauschenBot/1.1 ( mail address ) -
		-
	Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 de-de -
		image/..
		application/json
		text/..
		-
	FAST Enterprise Crawler 6 used by contosoa.com ( mail address ) - -
		text/..
	MerlBot - -
		application/vnd.php.serialized
	AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
		application/json
	python-wikitools/1.2 (User:LaraBot) -
		application/x-www-form-urlencoded
	Mozilla 5.0 (Apibot 0.30b5) - -
		application/vnd.php.serialized
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
		text/..
		application/xml
	IsraBot -
		-
	NoyaBot - -
		text/..
	Wikibot/2.0.1 CFNetwork/609 Darwin/13.0.0 en-us -
		image/..
		application/json
		text/..
	Robots.txt finder - -
		text/..
	GoogleBot - -
		text/..
	Wikibot/2.0.2 CFNetwork/609 Darwin/13.0.0 zh-cn -
		image/..
		application/json
		text/..
	Local Site Parser 1.0 en-us,en;q=0.5 -
		text/..
	Mozilla/5.0 (Bgbot 0.5) - -
		text/..
	OpenLink Virtuoso RDF crawler - -
		image/..
		text/..
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.2/1.8.5.168;) Opera Mini/3.1 - -
		image/..
		text/..
		-
	bitlybot - -
		text/..
		image/..
		-
	MediaWiki::PerlBot/User:Wdwdbot - -
		application/json
	bot: fr-anal - -
		application/json
	MediaWiki::Bot 3.1.5 -
		application/x-www-form-urlencoded
	AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
		application/json
	MediaWiki::Bot 3.1.5 - -
		application/json
	SoxBot PHP -
		-
	Inlibris.com XMLBot/1.0 - -
		text/..
	EarwigBot/0.2.dev.git4ff7612a (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) -
		application/x-www-form-urlencoded
	DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
		text/..
		application/xml
39687.77	total