Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Mar 2012 - 31 Mar 2012 (last 12 months)

This analysis is based on a 1:1000 sampled server log (squids)

See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot
For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'
In total 69,531,030 page requests (mime type text/html only!) per day are considered crawler requests, out of 468,517,840 external requests, which is 14.8%

Page requests for crawlers that specify a url in the agent string

Count
x 1000 Secondary domain
(~site) name URL Mime type User agent

google

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html - Mozilla/5.0 (compatible; GoogleBot/2.1; url)

desktop.google.com/ application/xml Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)

www.google.com/bot.html text/.. Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html - Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html image/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html text/.. DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

desktop.google.com/ image/.. Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)

www.google.com/bot.html - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

desktop.google.com/ text/.. Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)

www.google.com/feedfetcher.html image/.. Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

www.google.com/bot.html text/.. SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/feedfetcher.html - FeedFetcher-Google; (url)

www.google.com/feedfetcher.html application/xml FeedFetcher-Google; (url)

code.google.com/appengine application/json AppEngine-Google; (url; appid: s~redconceptual)

www.google.com/feedfetcher.html application/json Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

www.google.com/feedfetcher.html text/.. Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ortografia4)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ortopedianew)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wikien4)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wikien3)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: rarplayer)

code.google.com/p/crawler4j/ text/.. crawler4j (url)

www.google.com/feedfetcher.html text/.. FeedFetcher-Google; (url)

www.google.com/feedfetcher.html application/xml Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

www.google.com/bot.html - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)

code.google.com/appengine image/.. AppEngine-Google; (url; appid: s~senchaiosrc)

www.google.com/coop/cse/cref text/.. FeedFetcher-Google-CoOp; (url)

code.google.com/appengine application/json Mozilla 4.0 AppEngine-Google; (url; appid: prfleme)

code.google.com/appengine application/xml AppEngine-Google; (url; appid: wikipedia-raw)

code.google.com/appengine text/.. WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: myproxywx)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: abdulfat)

code.google.com/appengine application/json AppEngine-Google; (url; appid: prfleme)

desktop.google.com/ application/xml Mozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)

www.google.com/bot.html text/.. GoogleBot/2.1 (url)

code.google.com/appengine text/.. Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)

docs.google.com image/.. Mozilla/5.0 (compatible; GoogleDocs; documents; url)

code.google.com/appengine application/json MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: usawebdl)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: moveable-weather)

www.google.com/bot.html image/.. DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: azamasmadi)

code.google.com/appengine image/.. AppEngine-Google; (url; appid: d24-img)

code.google.com/appengine text/.. www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: pakgalaxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: boxapp)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kbworld24)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlp8)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlp6)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~keytanwiki3)

code.google.com/p/ronda application/json Ronda - url

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~keytanwiki4)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kires-roxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: simple-tools2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ridemyhell)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~keytanwiki2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxy-devakishor)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: atxproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxyusing121)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: pox)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~drizzlprox)

www.google.com/feedfetcher.html text/.. Mozilla/5.0 (compatible) FeedFetcher-Google;(url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webproxy8-9)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: simple-tools6)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: 100thpriest)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: usawebdl3)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: jptaravellahighschool)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: d24-img)

www.google.com/feedfetcher.html - Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

docs.google.com image/.. Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: argim-free)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tusawebproxy4)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: nashimlive-nashimnx)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wwwwebp2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: threewiki)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~deutiki)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wagagate)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: vebproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: quigonjinn03)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: zabastan)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: web-phpproxy)

desktop.google.com/ - Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~secure-facebook-ranney)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxynaungnaung)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~misterhac)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ilovethemothernature)

desktop.google.com/ image/.. Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: hideproxyz)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: cmd-proxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kaveriselvaraj)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webponline5)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: openeyeproxy)

www.google.com/feedfetcher.html image/.. Mozilla/5.0 (compatible) FeedFetcher-Google;(url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: nation4india)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: freeoursouls)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: raja584sekhar)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ivankrisproxyserver)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: dkoxyserv)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: sjbrundage123456789)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webproxy8-5)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~tpbitalia)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ikaryse)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: 114proxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: d-spark)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webponline9)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: web-proxy-hh)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~proxyseekkety)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: prexyproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: imabouncytigger)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kikopea-openproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: varlopie)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tinkernutsearch)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~nyinayminproxy2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webproxy8-2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: web4proxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: zmomyanmar)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~link123451)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: drrkproxxxy)

www.google.com/bot.html NONE/wikipedia - Mozilla/5.0 (compatible; GoogleBot/2.1; url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~kushgenius)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wmhsonline)

code.google.com/appengine text/.. Mozilla/5.0 (Windows; Windows NT 6.1; zh-CN; rv:1.9.2.2) AppEngine-Google; (url; appid: fwall-w15)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ivegotalovelybunch)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: python-proxy-server)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlq0)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: klobotdoor)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: easypox)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: semeout)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: worldwide-propaganda)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~aurora-prox)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: taterproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlq8)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: weps005)

docs.google.com image/.. Mozilla/5.0 (compatible; GoogleDocs; url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: weps003)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tacomansterprox)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: dustbunnytycoonmonitor)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: cdeskinsp)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: toom16-10)

code.google.com/appengine image/.. Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: nicoflysurf)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: aero-proxy)

www.google.com/feedfetcher.html image/.. FeedFetcher-Google; (url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadl6)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: gaucho-labnol)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: clh-sw)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tusawebproxy8)

code.google.com/appengine text/.. Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: pazvantoff)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: default-testing)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxygeekcoke)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: eclair4151)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: simple-tools9)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tusawebproxy0)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: weps002)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webponline8)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tgbeeson)

code.google.com/appengine text/.. oohEmbed.com AppEngine-Google; (url; appid: vipoembed)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: thakurproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlq5)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webponline0)

facebook

www.facebook.com/externalhit_uatext.php image/.. facebookexternalhit/1.0 (url)

www.facebook.com/externalhit_uatext.php text/.. facebookexternalhit/1.0 (url)

www.facebook.com/externalhit_uatext.php text/.. facebookexternalhit/1.1 (url)

developers.facebook.com image/.. facebookplatform/1.0 (url)

www.facebook.com/externalhit_uatext.php - facebookexternalhit/1.0 (url)

www.facebook.com/externalhit_uatext.php image/.. facebookexternalhit/1.1 (url)

developers.facebook.com - facebookplatform/1.0 (url)

www.facebook.com/externalhit_uatext.php - facebookexternalhit/1.1 (url)

developers.facebook.com text/.. facebookplatform/1.0 (url)

bing

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm - Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm application/vnd.php.serialized Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5

www.bing.com/bingbot.htm image/.. Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm text/.. User-Agent :Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) (via babelfish.yahoo.com)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: 546fga)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b4

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy0)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf712)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxy2fly6)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: bereturning)

google?

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html - Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html text/.. GoogleBot/2.1 (url)

www.google.com/bot.html application/vnd.php.serialized Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html image/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html text/.. DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html - Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html text/.. Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url) [UsableNet Lift Mobile]

www.google.com/bot.html application/xml Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

naver

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url)

help.naver.com/robots/ - Yeti/1.0 (NHN Corp.; url)

help.naver.com/robots/ image/.. Yeti/1.0 (NHN Corp.; url)

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3

baidu

www.baidu.com/search/spider.html text/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.html - Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.htm text/.. Baiduspider-image(url)

www.baidu.com/search/spider.html application/vnd.php.serialized Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.htm text/.. Baiduspider(url)

www.baidu.com/search/spider.html image/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.html application/xml Mozilla/5.0 (compatible; Baiduspider/2.0; url)

yahoo

help.yahoo.com/help/us/ysearch/slurp image/.. Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Slurp; url)

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. 'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'

listing.yahoo.co.jp/support/faq/int/other/other_001.html text/.. Y!J-BRJ/YATS crawler (url)

developer.yahoo.com/yql/provider text/.. Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2

help.yahoo.com/help/us/ysearch/slurp - Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)

help.yahoo.com/help/us/ysearch/slurp - Mozilla/5.0 (compatible; Yahoo! Slurp; url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html image/.. 'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'

help.yahoo.com/help/us/ysearch/slurp application/vnd.php.serialized Mozilla/5.0 (compatible Yahoo! Slurp/3.0 url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. Y!J-BRI/0.0.1 crawler ( url )

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. Y!J-BRT/1.0 crawler (url)

help.yahoo.com/help/us/ysearch/slurp application/json Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html - DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html text/.. DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; url)

yandex

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexBot/3.0; url)

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexDirect/3.0; url)

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexImages/3.0; url)

yandex.com/bots - Mozilla/5.0 (compatible; YandexBot/3.0; url)

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexImages/3.0; url)

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)

yandex.com/bots application/vnd.php.serialized Mozilla/5.0 (compatible; YandexBot/3.0; url)

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexBot/3.0; url)

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexNewslinks; url)

msn

search.msn.com/msnbot.htm text/.. msnbot/2.0b (url)._

search.msn.com/msnbot.htm text/.. msnbot-media/1.1 (url)

search.msn.com/msnbot.htm text/.. msnbot/2.0b (url)

search.msn.com/msnbot.htm text/.. msnbot-NewsBlogs/2.0b (url)

search.msn.com/msnbot.htm image/.. msnbot-media/1.1 (url)

search.msn.com/msnbot.htm text/.. msnbot-Products/1.0 (url)

search.msn.com/msnbot.htm text/.. msnbot-UDiscovery/2.0b (url)

search.msn.com/msnbot.htm text/.. msnbot/0.01 (url)

search.msn.com/msnbot.htm - msnbot/2.0b (url)

search.msn.com/msnbot.htm - msnbot-media/1.1 (url)

wwwgogetpapers

wwwgogetpapers.com/ application/json User-Agent: GoGetPapersBot (url)

wwwgogetpapers.com/ text/.. User-Agent: GoGetPapersBot (url)

sblog

fulltext.sblog.cz/screenshot/ image/.. Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)

fulltext.sblog.cz/ text/.. SeznamBot/3.0 (url)

fulltext.sblog.cz/screenshot/ text/.. Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)

fulltext.sblog.cz/ - SeznamBot/3.0 (url)

majestic12

www.majestic12.co.uk/bot.php? text/.. Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)

archive

www.archive.org/details/archive.org_bot text/.. Mozilla/5.0 (compatible; archive.org_bot url)

archive.org/details/archive.org_bot image/.. Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)

www.archive.org/details/archive.org_bot image/.. Mozilla/5.0 (compatible; archive.org_bot url)

www.archive.org/details/archive.org_bot - Mozilla/5.0 (compatible; archive.org_bot url)

archive.org/details/archive.org_bot text/.. Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)

php

pear.php.net/ application/vnd.php.serialized PEAR HTTP_Request class ( url )

pear.php.net/ application/xml PEAR HTTP_Request class ( url )

pear.php.net/package/http_request2 text/.. HTTP_Request2/0.5.2 (url) PHP/5.2.17

pear.php.net/ text/.. PEAR HTTP_Request class ( url )

pear.php.net/ image/.. PEAR HTTP_Request class ( url )

pear.php.net/package/http_request2 application/xml HTTP_Request2/2.0.0 (url) PHP/5.3.8

pear.php.net/package/http_request2 text/.. HTTP_Request2/2.0.0 (url) PHP/5.3.2-1ubuntu4.10

echonest

the.echonest.com/reader/ application/xml nestReader/0.3 (discovery; url; reader at echonest.com)

the.echonest.com/reader/ text/.. nestReader/0.3 (discovery; url; reader at echonest.com)

wordpress

danielradiorock.wordpress.com text/.. WordPress/3.4-alpha-20205; url

josefboberg.wordpress.com text/.. WordPress/3.4-alpha-20264; url

pennylibertygbow.wordpress.com text/.. WordPress/3.4-alpha-19994; url

josefboberg.wordpress.com text/.. WordPress/3.4-alpha-20150; url

klausgauger.wordpress.com text/.. WordPress/3.4-alpha-20150; url

driwancybermuseum.wordpress.com text/.. WordPress/3.4-alpha-20205; url

klausgauger.wordpress.com text/.. WordPress/3.4-alpha-20264; url

josefboberg.wordpress.com text/.. WordPress/3.4-alpha-20205; url

soso

help.soso.com/webspider.htm text/.. Sosospider(url)

help.soso.com/webspider.htm - Sosospider(url)

help.soso.com/webspider.htm application/xml Sosospider(url)

www.

www. text/.. GoogleBot/2.1 ( urlGoogleBot.com/bot.html)

www. text/.. GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)

www. - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)

www. text/.. GoogleBot/2.1 (urlGoogleBot.com/bot.html)

www. - GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)

youdao

www.youdao.com/help/webmaster/spider/ text/.. Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )

www.youdao.com/help/webmaster/spider/ - Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )

www.youdao.com/help/webmaster/spider/ image/.. Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)

www.youdao.com/help/webmaster/spider/ text/.. Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)

toolbar.youdao.com/ image/.. Youdao Toolbar (url)

www.youdao.com/help/webmaster/spider/ application/vnd.php.serialized Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )

www.youdao.com/help/webmaster/spider/ text/.. Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ),gzip(gfe) (via translate.google.com)

www.youdao.com/help/webmaster/spider/ image/.. Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )

sogou

www.sogou.com/docs/help/webmasters.htm#07 text/.. Sogou web spider/4.0(url)

www.sogou.com/docs/help/webmasters.htm#07 - Sogou web spider/4.0(url)

www.sogou.com/docs/help/webmasters.htm#07 application/vnd.php.serialized Sogou web spider/4.0(url)

www.sogou.com/docs/help/webmasters.htm#07 text/.. Sogou inst spider/4.0(url)

yacy

yacy.net/bot.html text/.. yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url

yacy.net/bot.html text/.. yacybot (sciencenet-any; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_30; Asia/ja) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-38-generic; java 1.7.0; America/en) url

yacy.net/bot.html text/.. yacybot (sciencenet/any; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; America/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url

yacy.net/bot.html text/.. yacybot (sciencenet/any; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) url

yacy.net/bot.html text/.. yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.42.9-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_01; Asia/ja) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32.36-228-scalaxy; java 1.6.0_18; Etc/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 3.0.0-ck1-solusos; java 1.6.0_18; GMT01:00/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 3.2.9-1-pae; java 1.7.0_03-icedtea; Asia/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 3.0.0-16-generic-pae; java 1.6.0_23; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 3.2.8-1-ARCH; java 1.7.0_03-icedtea; Asia/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) url

yacy.net/bot.html - yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 3.2.0-2-686-pae; java 1.6.0_24; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 3.0.0-ck1-solusos; java 1.6.0_26; GMT01:00/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.0-17-generic; java 1.7.0_03-icedtea; America/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.7-1.fc16.x86_64; java 1.7.0_b147-icedtea; Europe/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.41.10-3.fc15.x86_64; java 1.6.0_22; W-SU/ru) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 2.6.32-220.4.2.el6.i686; java 1.6.0_22; US/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.0.0-16-server; java 1.6.0_23; America/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_29; GMT/is) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 3.2.12-1-ARCH; java 1.7.0_03-icedtea; GB-Eire/en) url

yacy.net/bot.html text/.. yacybot (sciencenet/any; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.0.0-17-server; java 1.6.0_23; America/en) url

jike

shoulu.jike.com/spider.html text/.. Mozilla/5.0 (compatible; JikeSpider; url)

shoulu.jike.com/spider.html - Mozilla/5.0 (compatible; JikeSpider; url)

exabot

www.exabot.com/go/robot text/.. Mozilla/5.0 (compatible; Exabot/3.0; url)

www.exabot.com/go/robot - Mozilla/5.0 (compatible; Exabot/3.0; url)

ahrefs

ahrefs.com/robot/ text/.. Mozilla/5.0 (compatible; AhrefsBot/2.0; url)

ahrefs.com/robot/ - Mozilla/5.0 (compatible; AhrefsBot/2.0; url)

wikipedia

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.19.0 url

en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation text/.. WikiCleaner (url)

en.wikipedia.org/wiki/Web_crawler text/.. GoogleBot/Nutch-1.0 (Prototype; url; mail address )

ko.wikipedia.org text/.. url

fr.wikipedia.org/wiki/Utilisateur:Salebot application/json Salebot, see url (uses Perl MediaWiki::API)

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.19 url

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.18.0 url

en.wikipedia.org/wiki/Web_crawler - GoogleBot/Nutch-1.0 (Prototype; url; mail address )

crikey

blogs.crikey.com.au/game-on text/.. WordPress/3.2.1; url

discoveryengine

discoveryengine.com/discobot.html text/.. Mozilla/5.0 (compatible; discobot/2.0; url)

discoveryengine.com/discobot.html image/.. Mozilla/5.0 (compatible; discobot/2.0; url)

cibra

cibra.de/ text/.. CiBra Data Collector (url)

toolserver

wiki.toolserver.org/view/GeoHack text/.. Geohack (url)

toolserver.org/~dispenser/ text/.. DispensersTools (url)

toolserver.org/~dispenser/ application/json DispensersTools (url)

toolserver.org/~para/cgi-bin/kmlexport text/.. url libwww-perl/6.02

edu:8080

vancouver.cs.washington.edu:8080/ text/.. Mozilla/5.0/heritrix/3.1.0 (compatible;; url)

entireweb

www.entireweb.com/about/search_tech/speedy_spider/ text/.. Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)

mediawiki

www.mediawiki.org/ text/.. MediaWiki OAI Harvester 0.2 (url)

www.mediawiki.org/ text/.. MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)

80legs

www.80legs.com/webcrawler.html text/.. Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620

www.80legs.com/webcrawler.html image/.. Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620

enotes

www.enotes.com image/.. eNotesBot 2.0 (url)

www.enotes.com text/.. eNotesBot 2.0 (url)

gnip

www.gnip.com/ text/.. UnwindFetchor/1.0 (url)

www.gnip.com/ image/.. UnwindFetchor/1.0 (url)

wikidict

www.wikidict.de text/.. url

zum

help.zum.com/inquiry text/.. ZumBot/1.0 (ZUM Search; url)

help.zum.com/inquiry image/.. ZumBot/1.0 (ZUM Search; url)

bin-co

www.bin-co.com/php/scripts/load/ text/.. BinGet/1.00.A (url)

www.bin-co.com/php/scripts/load/ application/vnd.php.serialized BinGet/1.00.A (url)

flipboard

flipboard.com/browserproxy image/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)

flipboard.com/browserproxy text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)

flipboard.com/browserproxy text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)

flipboard.com/browserproxy application/json Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)

commoncrawl

www.commoncrawl.org/bot.html text/.. CCBot/1.0 (url)

lipperhey

www.lipperhey.com/ text/.. Mozilla/5.0 (compatible; Lipperhey Site Explorer; url)

sf

liferea.sf.net/ text/.. Liferea/0.x.x (Linux; en_US.UTF-8; url)

magpierss.sf.net text/.. MagpieRSS/0.7x (url)

liferea.sf.net/ text/.. Liferea/1.x.x (Linux; es_ES.UTF-8; url)

magpierss.sf.net application/xml MagpieRSS/0.72 (url; No cache)

traslated

mymemory.traslated.net/doc/ text/.. Mozilla/5.0 (MyMemory Bot url)

FeedBurner

www.FeedBurner.com text/.. FeedBurner/1.0 (url)

enwp

enwp.org/User:SDPatrolBot text/.. SDPatrolBot (url)

enwp.org/User:KingpinBot text/.. KingpinBot (url)

enwp.org/User:H3llkn0wz/WikiSharpAPI text/.. WikiSharpAPI/0.3 url (C# .NET)

wesee

www.wesee.com/en/support/bot/ image/.. WeSEE:Search/0.1 (Alpha, url)

www.wesee.com/en/support/bot/ text/.. WeSEE:Search/0.1 (Alpha, url)

daum

ws.daum.net/aboutWebSearch.html text/.. Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0

tab.search.daum.net/aboutWebSearch.html text/.. Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0

federatedmedia

federatedmedia.net text/.. Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1

federatedmedia.net image/.. Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1

kosmix

www.kosmix.com/html/kosmos.html application/xml Mozilla/5.0(compatible;Kosmos/1.0;url)

goo

help.goo.ne.jp/contact/ text/.. goo wikipedia (url)

help.goo.ne.jp/door/crawler.html text/.. ichiro/3.0 (url)

github

github.com/pauldix/typhoeus/tree/master text/.. Typhoeus - url

github.com/NeilCrosby/wikislurp application/vnd.php.serialized WikiSlurp (url)

github.com/edsu/linkypedia application/json linkpyediabot v0.1: url

freebase

www.freebase.com text/.. metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)

avantbrowser

www.avantbrowser.com text/.. Avant Browser (url)

www.avantbrowser.com text/.. Advanced Browser (url)

feedshow

www.feedshow.com text/.. FeedshowOnline (url)

www.feedshow.com text/.. Feedshow/x.0 (url; 1 subscriber)

newsgator

www.newsgator.com/ text/.. FeedDemon/2.7 (url; Microsoft Windows XP)

www.newsgator.com text/.. NewsGatorOnline/2.0 (url; 1 subscribers)

jetbrains

www.jetbrains.com/omea_reader/ text/.. JetBrains Omea Reader 1.0.x (url)

www.jetbrains.com/omea_reader/ text/.. JetBrains Omea Reader 2.0 Release Candidate 1 (url)

ephorus

www.ephorus.com/ text/.. Mozilla/5.0 (compatible; Ephorusbot/1.0.0; url)

netseer

www.netseer.com/crawler.html text/.. Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )

veveo

corporate.veveo.net/webmasters.html text/.. Mozilla/5.0 (compatible; Veveobot; url)

whatrhymeswith

www.whatrhymeswith.com/site/rhyme-bot text/.. RhymeBot/0.1 (url)

emining

emining.jp/ text/.. emBot-GalaBuzz/Nutch-1.0 (url; mail address )

wikimpress

wikimpress.org/ text/.. Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0

apache

lucene.apache.org/nutch/bot.html text/.. NutchCVS/0.7.2 (Nutch; url; mail address )

archive-it

archive-it.org/files/site-owners.html image/.. Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)

archive-it.org/files/site-owners.html text/.. Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)

archive-it.org/files/site-owners.html - Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)

yioop

www.yioop.com/bot.php text/.. Mozilla/5.0 (compatible; YioopBot; url)

www.yioop.com/bot.php image/.. Mozilla/5.0 (compatible; YioopBot; url)

accelobot

www.accelobot.com text/.. Mozilla/5.0 (compatible; heritrix/1.14.3 url)

bibalex

archive.bibalex.org/bot/ image/.. Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)

archive.bibalex.org/bot/ text/.. Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)

thearchangelmichael

thearchangelmichael.net text/.. WordPress/3.0; url

thearchangelmichael.net text/.. WordPress/3.3.1; url

info.thearchangelmichael.net text/.. WordPress/3.3.1; url

apercite

www.apercite.fr/robot/index.html image/.. Mozilla/5.0 (compatible; Apercite; url)

cmu

boston.lti.cs.cmu.edu/crawler_12/ text/.. Mozilla/5.0 (compatible; lemurwebcrawler mail address ; url)

hatena

a.hatena.ne.jp/help text/.. Hatena Antenna/0.5 (url)

wikiglass

wikiglass.com text/.. url : mail address

scoutjet

www.scoutjet.com/ text/.. Mozilla/5.0 (compatible; ScoutJet; url)

bsurprised

bsurprised.com/ text/.. BSurprised WikiBox 0.1.3 (url)

SearchNearMe

SearchNearMe.com/contact.php application/vnd.php.serialized SearchNearMe (url)

SearchNearMe.com/contact.php text/.. SearchNearMe (url)

tweetmeme

tweetmeme.com/ text/.. Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)

easybib

content.easybib.com/autocite/ text/.. EasyBib AutoCite (url)

content.easybib.com/autocite/ application/json EasyBib AutoCite (url)

xbmc

www.xbmc.org image/.. XBMC/11.0-RC2 Git:20120229-f38655f (iOS; 11.0.0 AppleTV2,1, Version 5.0.1 (Build 9A406a); url)

www.xbmc.org image/.. XBMC/11.0-RC2 Git:20120229-f38655f (Windows NT 6.1;WOW64;Win64;x64; url)

rssbandit

www.rssbandit.org text/.. RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )

tinyurl

tinyurl.com/64t5n text/.. Rome Client (url) Ver: 0.9

zipcommander

www.zipcommander.com/ text/.. 1st ZipCommander (Net) - url

nemui

mozshot.nemui.org/ text/.. Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)

ponderer

ponderer.org/download/annotate_google.user.js text/.. annotate_google; url

zootycoon

www.zootycoon.com text/.. Zoo Tycoon 2 Client -- url

timewe

timewe.net text/.. CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0

rssreader

www.rssreader.com text/.. RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0

orcabrowser

www.orcabrowser.com text/.. Orca Browser (url)

seebot

seebot.org text/.. Lynx/2.8 (;url)

graemef

graemef.com text/.. NewsGator FetchLinks extension/0.2.0 (url)

blogbridge

www.blogbridge.com/ text/.. BlogBridge 2.13 (url)

winpodder

winpodder.com text/.. WinPodder (url)

speaktoit

www.speaktoit.com application/json Speaktoit url

it-influentials

search.it-influentials.com/bot.htm text/.. Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)

warebay

www.warebay.com/bot.html text/.. Mozilla/5.0 (compatible; WBSearchBot/1.1; url)

kalooga

kalooga.com/crawler image/.. Mozilla/5.0 (compatible; KaloogaBot; url)

kalooga.com/crawler text/.. Mozilla/5.0 (compatible; KaloogaBot; url)

kula

kula.jp/endo text/.. endo/1.0 (Mac OS X; ppc i386; url)

ranchero

ranchero.com/netnewswire/ text/.. NetNewsWire/2.x (Mac OS X; url)

feeds4all

www.feeds4all.com/feedzcollector text/.. FeedZcollector v1.x (Platinum) url

snarfware

www.snarfware.com/ text/.. Snarfer/0.x.x (url)

sentymetr

sentymetr.pl/bot.html application/json Mozilla/5.0 (compatible; SentymetrBot 1.0; url)

sentymetr.pl/bot.html text/.. Mozilla/5.0 (compatible; SentymetrBot 1.0; url)

plagger

plagger.org/ text/.. Plagger/0.x.xx (url)

abonti

www.abonti.com text/.. Mozilla/5.0 (compatible; Abonti/0.91 - url)

superfeedr

superfeedr.com application/xml Superfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard

textdigger

textdigger.com text/.. Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1

blogscope

www.blogscope.net/ text/.. Mozilla/5.0 (compatible; BlogScope/1.0; url; U of Toronto)

ac

www.clips.ua.ac.be/pages/pattern application/json Pattern/1.0 url

www.clips.ua.ac.be/pages/pattern text/.. Pattern/1.0 url

www.clips.ua.ac.be/pages/pattern text/.. Pattern/2.3 url

www.clips.ua.ac.be/pages/pattern application/json Pattern/2.3 url

artez

www.artez.nl text/.. artezTest/0.1 (url)

picsearch

www.picsearch.com/bot.html text/.. psbot/0.1 (url)

kr:6600

www.checkprivacy.or.kr:6600/RS/PRIVACY_FAQ.jsp text/.. url

netnewswireapp

netnewswireapp.com/mac/ - NetNewsWire/3.3 (Mac OS X; url; gzip-happy)

linkbutler

www.linkbutler.de/spider text/.. lb-spider/Mozilla/5.0 Gecko/20100101 Firefox/10.0.2 (lb-spider; url; mail address )

tumblr

benderthewebrobot.tumblr.com text/.. Mozilla/5.0 (compatible; Bender; url)

rockpeaks

www.rockpeaks.com/contact text/.. RockPeaks/0.1 (url)

drupal

drupal.org/ text/.. User-Agent: Drupal (url)

drupal.org/ text/.. Drupal (url)

alexa

www.alexa.com/site/help/webmasters text/.. ia_archiver (url; mail address )

spinn3r

spinn3r.com/robot text/.. Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19

netvibes

www.netvibes.com text/.. Netvibes (url)

simplepie

simplepie.org application/xml SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103

simplepie.org text/.. SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103

Anonymouse

Anonymouse.org/ image/.. url (Unix)

Anonymouse.org/ text/.. url (Unix)

pagepeeker

pagepeeker.com/robots image/.. PagePeeker.com (info: url)

pagepeeker.com/robots text/.. PagePeeker.com (info: url)

bnf

www.bnf.fr/fr/outils/a.dl_web_capture_robot.html text/.. Mozilla/5.0 (compatible; bnf.fr_bot; url)

www.bnf.fr/fr/outils/a.dl_web_capture_robot.html image/.. Mozilla/5.0 (compatible; bnf.fr_bot; url)

ecrawlerdata

www.ecrawlerdata.com text/.. Mozilla/5.0 (compatible; heritrix/1.14.4 url)

metager2

metager2.de/technology.php text/.. Mozilla/5.0 (compatible; metager2-verification-bot; url)

topsy

labs.topsy.com/butterfly/ text/.. Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8

suggy

blog.suggy.com/was-ist-suggy/suggy-webcrawler/ text/.. Mozilla/5.0 (compatible; suggybot v0.01a, url)

duckduckgo

duckduckgo.com/duckduckpreview.html text/.. DuckDuckPreview/1.0; (url)

duckduckgo.com/duckduckpreview.html - DuckDuckPreview/1.0; (url)

duckduckgo.com/duckduckbot.html text/.. DuckDuckBot/1.1; (url)

plagiarismcheck

plagiarismcheck.org application/json WikiCrawl 1.0b (url contact-mail: mail address )

iis

www.iis.net/iisbot.html text/.. iisbot/1.0 (url)

turnitin

www.turnitin.com/robot/crawlerinfo.html text/.. TurnitinBot/2.1 (url)

paper

support.paper.li/entries/20023257-what-is-paper-li text/.. Mozilla/5.0 (compatible; PaperLiBot/2.1; url)

weblio

www.weblio.jp/ text/.. Mozilla/5.0 (compatible; WeblioBot; url)

froute

labs.froute.jp/pc2m/help.html text/.. Froute Mobile Gateway/1.0 (url)

fotopedia

www.fotopedia.com application/json Picor (url)

seegul

www.seegul.com text/.. Seegul ImageBot - url

instapaper

www.instapaper.com/ text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)

latestsearchtrends

latestsearchtrends.com text/.. WordPress/3.3.1; url

creativecommons

wiki.creativecommons.org/Metadata_Scraper text/.. CC Metadata Scaper url

ibis

ibis.ne.jp/browser/about.html image/.. Mozilla/4.0 (compatible; ibisBrowser; url)

ibis.ne.jp/browser/about.html text/.. Mozilla/4.0 (compatible; ibisBrowser; url)

searchtechnologies

www.searchtechnologies.com text/.. Mozilla/5.0 (compatible; heritrix/1.14.3 url)

pinterest

pinterest.com/ image/.. Pinterest/0.1 url

embed

support.embed.ly/ text/.. Mozilla/5.0 (compatible; Embedly/0.2; url)

support.embed.ly/ image/.. Mozilla/5.0 (compatible; Embedly/0.2; url)

semager

www.semager.de/blog/semager-bots/ text/.. Mozilla/5.0 (compatible; Semager/1.4c; url)

edu

ws.nju.edu.cn/falcons/ text/.. Mozilla/5.0 (compatible; Falconsbot; url)

sixxs

lb

www.lb.de text/.. lb-spider/Nutch-1.4 (lb-spider; url; mail address )

sonyericsson

www.sonyericsson.com/UAprof/R800xR301.xml image/.. Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1

104384.979999993 total

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 69,531,030 page requests (mime type text/html only!) per day are considered crawler requests, out of 468,517,840 external requests, which is 14.8%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	google
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	text/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/feedfetcher.html	-	FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/xml	FeedFetcher-Google; (url)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: s~redconceptual)
		www.google.com/feedfetcher.html	application/json	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortografia4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortopedianew)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: rarplayer)
		code.google.com/p/crawler4j/	text/..	crawler4j (url)
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/xml	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: s~senchaiosrc)
		www.google.com/coop/cse/cref	text/..	FeedFetcher-Google-CoOp; (url)
		code.google.com/appengine	application/json	Mozilla 4.0 AppEngine-Google; (url; appid: prfleme)
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw)
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: myproxywx)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: abdulfat)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: prfleme)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; documents; url)
		code.google.com/appengine	application/json	MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: moveable-weather)
		www.google.com/bot.html	image/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: azamasmadi)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pakgalaxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: boxapp)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kbworld24)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlp8)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlp6)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki3)
		code.google.com/p/ronda	application/json	Ronda - url
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kires-roxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: simple-tools2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ridemyhell)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxy-devakishor)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: atxproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxyusing121)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~drizzlprox)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webproxy8-9)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: simple-tools6)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: 100thpriest)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: jptaravellahighschool)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d24-img)
		www.google.com/feedfetcher.html	-	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: argim-free)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tusawebproxy4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nashimlive-nashimnx)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wwwwebp2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: threewiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~deutiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wagagate)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vebproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: quigonjinn03)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: zabastan)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web-phpproxy)
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~secure-facebook-ranney)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxynaungnaung)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~misterhac)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ilovethemothernature)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: hideproxyz)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: cmd-proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kaveriselvaraj)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline5)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: openeyeproxy)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nation4india)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: freeoursouls)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: raja584sekhar)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ivankrisproxyserver)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dkoxyserv)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: sjbrundage123456789)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webproxy8-5)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~tpbitalia)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ikaryse)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: 114proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d-spark)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline9)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web-proxy-hh)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~proxyseekkety)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: prexyproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: imabouncytigger)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kikopea-openproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: varlopie)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tinkernutsearch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~nyinayminproxy2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webproxy8-2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web4proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: zmomyanmar)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~link123451)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: drrkproxxxy)
		www.google.com/bot.html	NONE/wikipedia	- Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~kushgenius)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wmhsonline)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 6.1; zh-CN; rv:1.9.2.2) AppEngine-Google; (url; appid: fwall-w15)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ivegotalovelybunch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: python-proxy-server)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlq0)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: klobotdoor)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: easypox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: semeout)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: worldwide-propaganda)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~aurora-prox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: taterproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlq8)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: weps005)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: weps003)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tacomansterprox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: cdeskinsp)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: toom16-10)
		code.google.com/appengine	image/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nicoflysurf)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: aero-proxy)
		www.google.com/feedfetcher.html	image/..	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadl6)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: gaucho-labnol)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: clh-sw)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tusawebproxy8)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pazvantoff)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: default-testing)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxygeekcoke)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: eclair4151)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: simple-tools9)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tusawebproxy0)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: weps002)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline8)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tgbeeson)
		code.google.com/appengine	text/..	oohEmbed.com AppEngine-Google; (url; appid: vipoembed)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: thakurproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlq5)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline0)
	facebook
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
		developers.facebook.com	image/..	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.1 (url)
		developers.facebook.com	-	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.1 (url)
		developers.facebook.com	text/..	facebookplatform/1.0 (url)
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	application/vnd.php.serialized	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	User-Agent :Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) (via babelfish.yahoo.com)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: 546fga)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b4
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy0)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf712)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxy2fly6)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: bereturning)
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url) [UsableNet Lift Mobile]
		www.google.com/bot.html	application/xml	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider-image(url)
		www.baidu.com/search/spider.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider(url)
		www.baidu.com/search/spider.html	image/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	application/xml	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		listing.yahoo.co.jp/support/faq/int/other/other_001.html	text/..	Y!J-BRJ/YATS crawler (url)
		developer.yahoo.com/yql/provider	text/..	Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	image/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.com/help/us/ysearch/slurp	application/vnd.php.serialized	Mozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRI/0.0.1 crawler ( url )
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRT/1.0 crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html	-	DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html	text/..	DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; url)
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexDirect/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
		yandex.com/bots	application/vnd.php.serialized	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexNewslinks; url)
	msn
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)._
		search.msn.com/msnbot.htm	text/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-NewsBlogs/2.0b (url)
		search.msn.com/msnbot.htm	image/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-Products/1.0 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-UDiscovery/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot/0.01 (url)
		search.msn.com/msnbot.htm	-	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	-	msnbot-media/1.1 (url)
	wwwgogetpapers
		wwwgogetpapers.com/	application/json	User-Agent: GoGetPapersBot (url)
		wwwgogetpapers.com/	text/..	User-Agent: GoGetPapersBot (url)
	sblog
		fulltext.sblog.cz/screenshot/	image/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	-	SeznamBot/3.0 (url)
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url)
		archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
		www.archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; archive.org_bot url)
		www.archive.org/details/archive.org_bot	-	Mozilla/5.0 (compatible; archive.org_bot url)
		archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
	php
		pear.php.net/	application/vnd.php.serialized	PEAR HTTP_Request class ( url )
		pear.php.net/	application/xml	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/0.5.2 (url) PHP/5.2.17
		pear.php.net/	text/..	PEAR HTTP_Request class ( url )
		pear.php.net/	image/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	application/xml	HTTP_Request2/2.0.0 (url) PHP/5.3.8
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.0.0 (url) PHP/5.3.2-1ubuntu4.10
	echonest
		the.echonest.com/reader/	application/xml	nestReader/0.3 (discovery; url; reader at echonest.com)
		the.echonest.com/reader/	text/..	nestReader/0.3 (discovery; url; reader at echonest.com)
	wordpress
		danielradiorock.wordpress.com	text/..	WordPress/3.4-alpha-20205; url
		josefboberg.wordpress.com	text/..	WordPress/3.4-alpha-20264; url
		pennylibertygbow.wordpress.com	text/..	WordPress/3.4-alpha-19994; url
		josefboberg.wordpress.com	text/..	WordPress/3.4-alpha-20150; url
		klausgauger.wordpress.com	text/..	WordPress/3.4-alpha-20150; url
		driwancybermuseum.wordpress.com	text/..	WordPress/3.4-alpha-20205; url
		klausgauger.wordpress.com	text/..	WordPress/3.4-alpha-20264; url
		josefboberg.wordpress.com	text/..	WordPress/3.4-alpha-20205; url
	soso
		help.soso.com/webspider.htm	text/..	Sosospider(url)
		help.soso.com/webspider.htm	-	Sosospider(url)
		help.soso.com/webspider.htm	application/xml	Sosospider(url)
	www.
		www.	text/..	GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
		www.	-	GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
		www.	-	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	image/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		toolbar.youdao.com/	image/..	Youdao Toolbar (url)
		www.youdao.com/help/webmaster/spider/	application/vnd.php.serialized	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ),gzip(gfe) (via translate.google.com)
		www.youdao.com/help/webmaster/spider/	image/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	application/vnd.php.serialized	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou inst spider/4.0(url)
	yacy
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_30; Asia/ja) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-38-generic; java 1.7.0; America/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet/any; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet/any; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.42.9-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_01; Asia/ja) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32.36-228-scalaxy; java 1.6.0_18; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.0.0-ck1-solusos; java 1.6.0_18; GMT01:00/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.2.9-1-pae; java 1.7.0_03-icedtea; Asia/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.0.0-16-generic-pae; java 1.6.0_23; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.2.8-1-ARCH; java 1.7.0_03-icedtea; Asia/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) url
		yacy.net/bot.html	-	yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.2.0-2-686-pae; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.0.0-ck1-solusos; java 1.6.0_26; GMT01:00/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-17-generic; java 1.7.0_03-icedtea; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.7-1.fc16.x86_64; java 1.7.0_b147-icedtea; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.41.10-3.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 2.6.32-220.4.2.el6.i686; java 1.6.0_22; US/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-16-server; java 1.6.0_23; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_29; GMT/is) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 3.2.12-1-ARCH; java 1.7.0_03-icedtea; GB-Eire/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet/any; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.0-17-server; java 1.6.0_23; America/en) url
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url)
		shoulu.jike.com/spider.html	-	Mozilla/5.0 (compatible; JikeSpider; url)
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url)
		www.exabot.com/go/robot	-	Mozilla/5.0 (compatible; Exabot/3.0; url)
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
		ahrefs.com/robot/	-	Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.0 url
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WikiCleaner (url)
		en.wikipedia.org/wiki/Web_crawler	text/..	GoogleBot/Nutch-1.0 (Prototype; url; mail address )
		ko.wikipedia.org	text/..	url
		fr.wikipedia.org/wiki/Utilisateur:Salebot	application/json	Salebot, see url (uses Perl MediaWiki::API)
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19 url
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.18.0 url
		en.wikipedia.org/wiki/Web_crawler	-	GoogleBot/Nutch-1.0 (Prototype; url; mail address )
	crikey
		blogs.crikey.com.au/game-on	text/..	WordPress/3.2.1; url
	discoveryengine
		discoveryengine.com/discobot.html	text/..	Mozilla/5.0 (compatible; discobot/2.0; url)
		discoveryengine.com/discobot.html	image/..	Mozilla/5.0 (compatible; discobot/2.0; url)
	cibra
		cibra.de/	text/..	CiBra Data Collector (url)
	toolserver
		wiki.toolserver.org/view/GeoHack	text/..	Geohack (url)
		toolserver.org/~dispenser/	text/..	DispensersTools (url)
		toolserver.org/~dispenser/	application/json	DispensersTools (url)
		toolserver.org/~para/cgi-bin/kmlexport	text/..	url libwww-perl/6.02
	edu:8080
		vancouver.cs.washington.edu:8080/	text/..	Mozilla/5.0/heritrix/3.1.0 (compatible;; url)
	entireweb
		www.entireweb.com/about/search_tech/speedy_spider/	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
	mediawiki
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url)
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	image/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
	enotes
		www.enotes.com	image/..	eNotesBot 2.0 (url)
		www.enotes.com	text/..	eNotesBot 2.0 (url)
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
		www.gnip.com/	image/..	UnwindFetchor/1.0 (url)
	wikidict
		www.wikidict.de	text/..	url
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url)
		help.zum.com/inquiry	image/..	ZumBot/1.0 (ZUM Search; url)
	bin-co
		www.bin-co.com/php/scripts/load/	text/..	BinGet/1.00.A (url)
		www.bin-co.com/php/scripts/load/	application/vnd.php.serialized	BinGet/1.00.A (url)
	flipboard
		flipboard.com/browserproxy	image/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	application/json	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
	commoncrawl
		www.commoncrawl.org/bot.html	text/..	CCBot/1.0 (url)
	lipperhey
		www.lipperhey.com/	text/..	Mozilla/5.0 (compatible; Lipperhey Site Explorer; url)
	sf
		liferea.sf.net/	text/..	Liferea/0.x.x (Linux; en_US.UTF-8; url)
		magpierss.sf.net	text/..	MagpieRSS/0.7x (url)
		liferea.sf.net/	text/..	Liferea/1.x.x (Linux; es_ES.UTF-8; url)
		magpierss.sf.net	application/xml	MagpieRSS/0.72 (url; No cache)
	traslated
		mymemory.traslated.net/doc/	text/..	Mozilla/5.0 (MyMemory Bot url)
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url)
	enwp
		enwp.org/User:SDPatrolBot	text/..	SDPatrolBot (url)
		enwp.org/User:KingpinBot	text/..	KingpinBot (url)
		enwp.org/User:H3llkn0wz/WikiSharpAPI	text/..	WikiSharpAPI/0.3 url (C# .NET)
	wesee
		www.wesee.com/en/support/bot/	image/..	WeSEE:Search/0.1 (Alpha, url)
		www.wesee.com/en/support/bot/	text/..	WeSEE:Search/0.1 (Alpha, url)
	daum
		ws.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
		tab.search.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
	federatedmedia
		federatedmedia.net	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
		federatedmedia.net	image/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	kosmix
		www.kosmix.com/html/kosmos.html	application/xml	Mozilla/5.0(compatible;Kosmos/1.0;url)
	goo
		help.goo.ne.jp/contact/	text/..	goo wikipedia (url)
		help.goo.ne.jp/door/crawler.html	text/..	ichiro/3.0 (url)
	github
		github.com/pauldix/typhoeus/tree/master	text/..	Typhoeus - url
		github.com/NeilCrosby/wikislurp	application/vnd.php.serialized	WikiSlurp (url)
		github.com/edsu/linkypedia	application/json	linkpyediabot v0.1: url
	freebase
		www.freebase.com	text/..	metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
	avantbrowser
		www.avantbrowser.com	text/..	Avant Browser (url)
		www.avantbrowser.com	text/..	Advanced Browser (url)
	feedshow
		www.feedshow.com	text/..	FeedshowOnline (url)
		www.feedshow.com	text/..	Feedshow/x.0 (url; 1 subscriber)
	newsgator
		www.newsgator.com/	text/..	FeedDemon/2.7 (url; Microsoft Windows XP)
		www.newsgator.com	text/..	NewsGatorOnline/2.0 (url; 1 subscribers)
	jetbrains
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 1.0.x (url)
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 2.0 Release Candidate 1 (url)
	ephorus
		www.ephorus.com/	text/..	Mozilla/5.0 (compatible; Ephorusbot/1.0.0; url)
	netseer
		www.netseer.com/crawler.html	text/..	Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
	veveo
		corporate.veveo.net/webmasters.html	text/..	Mozilla/5.0 (compatible; Veveobot; url)
	whatrhymeswith
		www.whatrhymeswith.com/site/rhyme-bot	text/..	RhymeBot/0.1 (url)
	emining
		emining.jp/	text/..	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
	wikimpress
		wikimpress.org/	text/..	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
	apache
		lucene.apache.org/nutch/bot.html	text/..	NutchCVS/0.7.2 (Nutch; url; mail address )
	archive-it
		archive-it.org/files/site-owners.html	image/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
		archive-it.org/files/site-owners.html	text/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
		archive-it.org/files/site-owners.html	-	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
	yioop
		www.yioop.com/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot; url)
		www.yioop.com/bot.php	image/..	Mozilla/5.0 (compatible; YioopBot; url)
	accelobot
		www.accelobot.com	text/..	Mozilla/5.0 (compatible; heritrix/1.14.3 url)
	bibalex
		archive.bibalex.org/bot/	image/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
		archive.bibalex.org/bot/	text/..	Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
	thearchangelmichael
		thearchangelmichael.net	text/..	WordPress/3.0; url

Page requests for probable crawlers, recognized by keyword

Count
x 1000 Agent string

Mime type (count ≥ 3)

PythonWikipediaBot/1.0

application/json

application/xml

text/..

-

image/..

php wikibot classes

application/vnd.php.serialized

text/..

-

MediaWikiCrawler-Google/2.0 ( mail address )

text/..

-

GoogleBot-Image/1.0

image/..

text/..

-

LinkParser/2.0

text/..

MoovidaBot/0.1

text/..

Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )

text/..

-

application/pdf

GoogleBot-Image/1.0

text/..

image/..

-

application/vnd.php.serialized

application/json

wikiwix-bot-3.0

text/..

-

mail address

application/vnd.php.serialized

text/..

DotNetWikiBot/1.0

text/..

SemrushBot/0.91

text/..

image/..

application/ogg

-

video/ogg

ClueBot/1.1

application/vnd.php.serialized

-

Peachy MediaWiki Bot API Version 1.0

application/vnd.php.serialized

-

Answersbot

text/..

spider

text/..

application/vnd.php.serialized

application/json

application/xml

image/..

Pywikipediabot/2.0

application/json

K-Crawler

text/..

application/ogg

Mozilla/5.0 (compatible; Ezooms/1.0; mail address )

text/..

application/vnd.php.serialized

-

image/..

application/xml

ClueBot/2.0

application/vnd.php.serialized

Mozilla 5.0 (Apibot 0.32)

application/vnd.php.serialized

bob's crappy crawler; contact: mail address

text/..

-

DigitalsmithsBot

text/..

Mozilla/5.0 (compatible; YandexBot/3.0)

image/..

text/..

video/ogg

MediaWiki::Bot/3.2.6

application/json

-

AnomieBOT 1.0 (TagDater)

application/json

application/x-www-form-urlencoded

BritannicaProjBot mail address

text/..

python-wikitools/1.2 (User:BernsteinBot)

application/json

application/x-www-form-urlencoded

text/..

DotNetWikiBot/2.98 (Unix 3.0.0.12; )

text/..

application/xml

Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; mail address )

text/..

image/..

application/pdf

-

mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)

application/json

application/x-www-form-urlencoded

VeeloBot 1.0

text/..

-

image/..

application/ogg

wikbot/1.50 CFNetwork/548.1.4 Darwin/11.0.0

image/..

application/json

text/..

-

YBot/0.1

application/vnd.php.serialized

DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )

text/..

application/xml

image/..

MireoBot

text/..

-

application/xml

FAST Enterprise Crawler 6 used by ESP ( mail address )

text/..

Tawbot (public svn release; plwiki)

text/..

AnomieBOT 1.0 (ReplaceExternalLinks2)

application/json

daytrippy.com Crawler

application/json

text/..

wikbot/1.50 CFNetwork/548.0.4 Darwin/11.0.0

image/..

application/json

text/..

-

Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)

image/..

text/..

-

MLBot (www.metadatalabs.com/mlbot)

text/..

application/vnd.php.serialized

-

image/..

Test Webbot

text/..

SineBot/1.5.18(User:SineBot)

application/vnd.php.serialized

text/..

plantspedia data crawler

text/..

SchoolReviewNetworkWikiBot

application/json

DotNetWikiBot/2.97 (Unix 2.6.32.38; )

text/..

DotNetWikiBot/2.97 (Unix 5.10.0.0; )

application/xml

text/..

-

JavaCrawler/1.1

text/..

UCMore Crawler App

text/..

-

TrueKnowledgeBot bot mail address >

application/xml

application/vnd.php.serialized

Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot

text/..

-

AniBot/0.9 php/curl

application/vnd.php.serialized

SiteSeekerCrawler/1.0

text/..

Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9

text/..

-

Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)

text/..

DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; )

text/..

application/xml

COIBot/1.00

text/..

AnomieBOT 1.0 (FlagIconRemover)

application/json

CorenSearchBot/1.5 en libwww-perl/6.02

text/..

DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )

text/..

application/xml

Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.2.115) Opera Mini/3.1

image/..

-

text/..

FAST Enterprise Crawler 6 used by LexisNexis ( mail address )

text/..

-

Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )

text/..

-

application/xml

HTMLParser/2.0

text/..

My Nutch Spider/Nutch-1.4

text/..

image/..

application/pdf

HRoestBot, de-wikipedia using pywikipedia framework

text/..

application/json

application/xml

wikiparser/1 CFNetwork/454.12.4 Darwin/10.8.0 (x86_64) (MacPro5,1)

image/..

text/..

GoogleBot 2.1

text/..

DotNetWikiBot/2.96 (Unix 5.10.0.0; )

text/..

application/xml

Webwiki Search Engine Bot - www.webwiki.de

text/..

SiocWikiBot/1.0

application/vnd.php.serialized

text/..

AnomieBOT 1.0 (OrphanReferenceFixer)

application/json

AnomieBOT 1.0 (TemplateSubster)

application/json

TheKeens bot

text/..

-

Wikibot

text/..

image/..

-

DotNetWikiBot/2.97 (Unix 2.6.32.36; )

text/..

bitlybot

text/..

image/..

-

GSLFbot

text/..

application/xml

Twitterbot/1.0

text/..

image/..

AnomieBOT 1.0 (BAGBot)

application/json

text/..

SurakWare MediaWiki Bot/1.0

text/..

XLinkBot/1.00

text/..

~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )

text/..

GNAA-bot

text/..

OrlodrimBot/1.0

text/..

TVersity Media Robot

text/..

Opera/8.01 (J2ME/MIDP; MXit WebBot/1.7.7.93) Opera Mini/3.1

image/..

-

text/..

wikbotlite/1.50 CFNetwork/548.1.4 Darwin/11.0.0

image/..

application/json

text/..

super cool bot

application/vnd.php.serialized

application/json

FAST Enterprise Crawler/5.3.4 ( mail address )

text/..

-

Magus Bot 1.0

text/..

image/..

DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )

text/..

application/xml

python-wikitools/1.2 (User:Mr.Z-bot)

application/json

AnomieBOT 1.0 (PERTableUpdater)

application/json

text/..

Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9

image/..

text/..

DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )

text/..

application/xml

image/..

HosiryuhosiBot IRC-RecentChanges Util

text/..

-

LinksCrawler 0.1beta

text/..

-

Nutch-Spider/Nutch-1.4

text/..

Goalkeeperbot(User:Beetstra)/1.0

text/..

DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )

text/..

application/xml

Wikibot 1.54 (Macintosh; Mac OS X 10.7.3; en_TW)

image/..

text/..

Baiduspider

text/..

Freebase Deathbot

text/..

wikbotlite/1.50 CFNetwork/548.0.4 Darwin/11.0.0

image/..

application/json

text/..

-

Local Site Parser 1.0

text/..

DotNetWikiBot/2.9 (Unix 5.10.0.0; )

text/..

Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)

text/..

-

wikbot/1.50 CFNetwork/485.13.9 Darwin/11.0.0

image/..

application/json

text/..

-

wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0

image/..

application/json

text/..

-

DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7600.0; )

text/..

Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.4.121) Opera Mini/3.1

image/..

-

text/..

CheMoBot/1.00

text/..

Empedia Bot

text/..

DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )

text/..

application/xml

FAST Search Web Crawler 14.0.0325.0000

text/..

-

KWSS Crawler Ver. 0.1

text/..

DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )

text/..

HTMLParser/1.6

text/..

AnomieBOT 1.0 (RandomPagePicker)

application/json

COIBot/2.0

text/..

HBC Archive Indexerbot 0.9a

text/..

WikiBot/0.1

text/..

application/xml

Mozilla 5.0 (Apibot 0.30b5)

application/vnd.php.serialized

Mozilla/5.0 (Bgbot 0.5)

text/..

Zing-BottaBot/1.0

text/..

Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.3.119) Opera Mini/3.1

image/..

-

text/..

OrangeCrawler/Nutch-1.0 ( mail address )

text/..

DNSTallyKwBot/0.2

text/..

MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)

text/..

application/vnd.php.serialized

-

MyCuteBot/0.1

text/..

application/json

application/vnd.php.serialized

Geni ircpybot 1.0

text/..

application/json

python-wikitools/1.2 (User:LaraBot)

application/json

My Bot

image/..

text/..

IssueCrawler

text/..

AnomieBOT 1.0 (AFDMergeFromCleaner)

application/json

Hexabot V1.3 - curl - api.php

text/..

GermCrawler

application/json

text/..

Mozilla/5.0 (compatible; UnisterBot; mail address )

text/..

MediaWiki::Bot 3.1.5

application/json

DotNetWikiBot/2.94 (Microsoft Windows NT 5.1.2600 Service Pack 3; )

text/..

application/xml

My Nutch Spider/Nutch-1.4 (JPO)

text/..

16158.28 total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

Generated on Mon, Aug 6, 2012 15:03
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	PythonWikipediaBot/1.0
		application/json
		application/xml
		text/..
		-
		image/..
	php wikibot classes
		application/vnd.php.serialized
		text/..
		-
	MediaWikiCrawler-Google/2.0 ( mail address )
		text/..
		-
	GoogleBot-Image/1.0
		image/..
		text/..
		-
	LinkParser/2.0
		text/..
	MoovidaBot/0.1
		text/..
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
		text/..
		-
		application/pdf
	GoogleBot-Image/1.0
		text/..
		image/..
		-
		application/vnd.php.serialized
		application/json
	wikiwix-bot-3.0
		text/..
		-
	mail address
		application/vnd.php.serialized
		text/..
	DotNetWikiBot/1.0
		text/..
	SemrushBot/0.91
		text/..
		image/..
		application/ogg
		-
		video/ogg
	ClueBot/1.1
		application/vnd.php.serialized
		-
	Peachy MediaWiki Bot API Version 1.0
		application/vnd.php.serialized
		-
	Answersbot
		text/..
	spider
		text/..
		application/vnd.php.serialized
		application/json
		application/xml
		image/..
	Pywikipediabot/2.0
		application/json
	K-Crawler
		text/..
		application/ogg
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
		text/..
		application/vnd.php.serialized
		-
		image/..
		application/xml
	ClueBot/2.0
		application/vnd.php.serialized
	Mozilla 5.0 (Apibot 0.32)
		application/vnd.php.serialized
	bob's crappy crawler; contact: mail address
		text/..
		-
	DigitalsmithsBot
		text/..
	Mozilla/5.0 (compatible; YandexBot/3.0)
		image/..
		text/..
		video/ogg
	MediaWiki::Bot/3.2.6
		application/json
		-
	AnomieBOT 1.0 (TagDater)
		application/json
		application/x-www-form-urlencoded
	BritannicaProjBot mail address
		text/..
	python-wikitools/1.2 (User:BernsteinBot)
		application/json
		application/x-www-form-urlencoded
		text/..
	DotNetWikiBot/2.98 (Unix 3.0.0.12; )
		text/..
		application/xml
	Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; mail address )
		text/..
		image/..
		application/pdf
		-
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
		application/json
		application/x-www-form-urlencoded
	VeeloBot 1.0
		text/..
		-
		image/..
		application/ogg
	wikbot/1.50 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	YBot/0.1
		application/vnd.php.serialized
	DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		image/..
	MireoBot
		text/..
		-
		application/xml
	FAST Enterprise Crawler 6 used by ESP ( mail address )
		text/..
	Tawbot (public svn release; plwiki)
		text/..
	AnomieBOT 1.0 (ReplaceExternalLinks2)
		application/json
	daytrippy.com Crawler
		application/json
		text/..
	wikbot/1.50 CFNetwork/548.0.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		image/..
		text/..
		-
	MLBot (www.metadatalabs.com/mlbot)
		text/..
		application/vnd.php.serialized
		-
		image/..
	Test Webbot
		text/..
	SineBot/1.5.18(User:SineBot)
		application/vnd.php.serialized
		text/..
	plantspedia data crawler
		text/..
	SchoolReviewNetworkWikiBot
		application/json
	DotNetWikiBot/2.97 (Unix 2.6.32.38; )
		text/..
	DotNetWikiBot/2.97 (Unix 5.10.0.0; )
		application/xml
		text/..
		-
	JavaCrawler/1.1
		text/..
	UCMore Crawler App
		text/..
		-
	TrueKnowledgeBot bot mail address >
		application/xml
		application/vnd.php.serialized
	Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
		text/..
		-
	AniBot/0.9 php/curl
		application/vnd.php.serialized
	SiteSeekerCrawler/1.0
		text/..
	Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
		text/..
		-
	Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
		text/..
	DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
	COIBot/1.00
		text/..
	AnomieBOT 1.0 (FlagIconRemover)
		application/json
	CorenSearchBot/1.5 en libwww-perl/6.02
		text/..
	DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.2.115) Opera Mini/3.1
		image/..
		-
		text/..
	FAST Enterprise Crawler 6 used by LexisNexis ( mail address )
		text/..
		-
	Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
		text/..
		-
		application/xml
	HTMLParser/2.0
		text/..
	My Nutch Spider/Nutch-1.4
		text/..
		image/..
		application/pdf
	HRoestBot, de-wikipedia using pywikipedia framework
		text/..
		application/json
		application/xml
	wikiparser/1 CFNetwork/454.12.4 Darwin/10.8.0 (x86_64) (MacPro5,1)
		image/..
		text/..
	GoogleBot 2.1
		text/..
	DotNetWikiBot/2.96 (Unix 5.10.0.0; )
		text/..
		application/xml
	Webwiki Search Engine Bot - www.webwiki.de
		text/..
	SiocWikiBot/1.0
		application/vnd.php.serialized
		text/..
	AnomieBOT 1.0 (OrphanReferenceFixer)
		application/json
	AnomieBOT 1.0 (TemplateSubster)
		application/json
	TheKeens bot
		text/..
		-
	Wikibot
		text/..
		image/..
		-
	DotNetWikiBot/2.97 (Unix 2.6.32.36; )
		text/..
	bitlybot
		text/..
		image/..
		-
	GSLFbot
		text/..
		application/xml
	Twitterbot/1.0
		text/..
		image/..
	AnomieBOT 1.0 (BAGBot)
		application/json
		text/..
	SurakWare MediaWiki Bot/1.0
		text/..
	XLinkBot/1.00
		text/..
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
		text/..
	GNAA-bot
		text/..
	OrlodrimBot/1.0
		text/..
	TVersity Media Robot
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.7.7.93) Opera Mini/3.1
		image/..
		-
		text/..
	wikbotlite/1.50 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
	super cool bot
		application/vnd.php.serialized
		application/json
	FAST Enterprise Crawler/5.3.4 ( mail address )
		text/..
		-
	Magus Bot 1.0
		text/..
		image/..
	DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	python-wikitools/1.2 (User:Mr.Z-bot)
		application/json
	AnomieBOT 1.0 (PERTableUpdater)
		application/json
		text/..
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
		image/..
		text/..
	DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
		image/..
	HosiryuhosiBot IRC-RecentChanges Util
		text/..
		-
	LinksCrawler 0.1beta
		text/..
		-
	Nutch-Spider/Nutch-1.4
		text/..
	Goalkeeperbot(User:Beetstra)/1.0
		text/..
	DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Wikibot 1.54 (Macintosh; Mac OS X 10.7.3; en_TW)
		image/..
		text/..
	Baiduspider
		text/..
	Freebase Deathbot
		text/..
	wikbotlite/1.50 CFNetwork/548.0.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	Local Site Parser 1.0
		text/..
	DotNetWikiBot/2.9 (Unix 5.10.0.0; )
		text/..
	Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
		text/..
		-
	wikbot/1.50 CFNetwork/485.13.9 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7600.0; )
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.4.121) Opera Mini/3.1
		image/..
		-
		text/..
	CheMoBot/1.00
		text/..
	Empedia Bot
		text/..
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
		text/..
		application/xml
	FAST Search Web Crawler 14.0.0325.0000
		text/..
		-
	KWSS Crawler Ver. 0.1
		text/..
	DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
		text/..
	HTMLParser/1.6
		text/..
	AnomieBOT 1.0 (RandomPagePicker)
		application/json
	COIBot/2.0
		text/..
	HBC Archive Indexerbot 0.9a
		text/..
	WikiBot/0.1
		text/..
		application/xml
	Mozilla 5.0 (Apibot 0.30b5)
		application/vnd.php.serialized
	Mozilla/5.0 (Bgbot 0.5)
		text/..
	Zing-BottaBot/1.0
		text/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.3.119) Opera Mini/3.1
		image/..
		-
		text/..
	OrangeCrawler/Nutch-1.0 ( mail address )
		text/..
	DNSTallyKwBot/0.2
		text/..
	MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
		text/..
		application/vnd.php.serialized
		-
	MyCuteBot/0.1
		text/..
		application/json
		application/vnd.php.serialized
	Geni ircpybot 1.0
		text/..
		application/json
	python-wikitools/1.2 (User:LaraBot)
		application/json
	My Bot
		image/..
		text/..
	IssueCrawler
		text/..
	AnomieBOT 1.0 (AFDMergeFromCleaner)
		application/json
	Hexabot V1.3 - curl - api.php
		text/..
	GermCrawler
		application/json
		text/..
	Mozilla/5.0 (compatible; UnisterBot; mail address )
		text/..
	MediaWiki::Bot 3.1.5
		application/json
	DotNetWikiBot/2.94 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	My Nutch Spider/Nutch-1.4 (JPO)
		text/..
16158.28	total