Wikimedia Traffic Analysis Report - Crawler requests

Daily averages, based on sample period: 1 May 2011 - 31 May 2011
 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 54,673,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 399,285,000 external requests, which is 13.7%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
13,488google
10,624 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
691 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
580 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
331 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
213 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
128 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
104 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
94 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
82 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
68 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
64 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
59 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
54 code.google.com/p/crawler4j/text/..crawler4j (url)
34 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
33 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
27 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
22 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
20 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
20 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
19 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
17 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
16 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
16 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
13 code.google.com/appengineapplication/jsonMozilla 3.5 AppEngine-Google; (url; appid: prfleme)
9 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
9 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
9 sites.google.com/site/bendercrawlertext/..Mozilla/5.0 (compatible; Bender; url)
9 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
8 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; url)
7 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: oohembed)
6 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
6 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findadvise)
4 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
4 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: retimeme)
4 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nwikiproxy)
3 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: nwikiproxy)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lullar-data),gzip(gfe) (via translate.google.com)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
11,901yahoo
8,770 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
2,710 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
125 misc.yahoo.com.cn/help.htmltext/..Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
48 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
44 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
41 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! DE Slurp; url)
30 help.yahoo.com/help/us/ysearch/crawling/crawling-01.htmltext/..Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; url)
17 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
16 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
16 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
15 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url)
12 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
12 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
10 misc.yahoo.com.cn/help.html-Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
9 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
6 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
6 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
5 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url)
3 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible; Yahoo! Slurp; url)
10,748facebook
5,824 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
4,456 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
307 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
122 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
30 developers.facebook.comimage/..facebookplatform/1.0 (url)
7 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
7,711bing
6,018 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
1,671 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
17 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
6,250google?
5,709 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
188 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
187 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
47 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
45 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
35 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
14 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
10 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
4 www.google.com/bot.htmltext/..Mozilla/5.0(compatible;GoogleBot/2.1;url)
4 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b3
2,261msn
1,545 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
282 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
105 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
105 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
102 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
99 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
7 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
5 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._ (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
4 search.msn.com/msnbot.htmapplication/xmlmsnbot/2.0b (url)._
3 search.msn.com/msnbot.htmtext/..User-Agent :msnbot/2.0b (url)._
2,156naver
2,082 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
42 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
10 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
9 help.naver.com/customer_webtxt_02.jsptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
6 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
1,847yandex
1,440 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
236 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
70 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
60 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
14 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
11 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
9 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url)
3 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
1,075baidu
672 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
178 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
115 www.baidu.jp/spider/text/..Baiduspider(url)
50 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
21 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
19 www.baidu.jp/spider/text/..DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0;url)
9 www.baidu.com/search/spider.htm-Baiduspider(url)
4 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
3 www.baidu.jp/spider/text/..BaiduImagespider(url)
396youdao
351 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
19 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
15 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
6 toolbar.youdao.com/image/..Youdao Toolbar (url)
3 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
357traslated
357 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
302entireweb
295 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
4 www.entireweb.com/about/search_tech/speedy_spider/-Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
299yacy
72 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) url
65 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-28-generic; java 1.6.0_20; Europe/en) url
23 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-31-generic; java 1.6.0_20; Europe/en) url
14 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_17; Etc/en) url
14 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) url
10 yacy.net/bot.htmltext/..yacybot (sciencenet/any; amd64 Linux 2.6.32-31-generic; java 1.6.0_20; Europe/en) url
8 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_20; Asia/en) url
7 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-28-server; java 1.6.0_20; Europe/en) url
6 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-32-generic; java 1.6.0_24; Europe/de) url
6 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-31-generic; java 1.6.0_20; Europe/en) url
6 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.18-194.32.1.el5.centos.plus; java 1.6.0_17; Europe/en) url
6 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/de) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.35-gentoo-r4; java 1.6.0_20; Europe/el) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) url
5 yacy.net/bot.htmltext/..yacybot (webportal-global; i386 Linux 2.6.32-24-generic-pae; java 1.6.0_20; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_24; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (sciencenet/any; amd64 Linux 2.6.32-30-generic; java 1.6.0_20; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_24; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) url
295sblog
181 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
42 fulltext.sblog.cz/text/..SeznamBot/3.0-beta (url)
32 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
19 fulltext.sblog.cz/robot/text/..SeznamBot/2.0 (url)
13 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
4 fulltext.sblog.cz/text/..SeznamBot/3.0-beta (url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
216majestic12
215 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
214exabot
184 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
20 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
9 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
207php
87 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
54 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
38 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
23 pear.php.net/text/..PEAR HTTP_Request class ( url )
200bsurprised
177 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
23 bsurprised.com/text/..BSurprised WikiBox 0.1 (url)
191wikipedia
53 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.13.0 url
51 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
39 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.11.0 url
17 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.10.0 url
11 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.12.0 url
6 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.1.0 url
5 en.wikipedia.orgtext/..url
3 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
191enwp
171 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
15 enwp.org/User:KingpinBottext/..KingpinBot (url)
4 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
170wordpress
33 driwancybermuseum.wordpress.comtext/..WordPress/MU; url
10 arthur2rcasc.wordpress.comtext/..WordPress/MU; url
8 kterrl.wordpress.comtext/..WordPress/MU; url
8 quantenheilungen.wordpress.comtext/..WordPress/MU; url
5 worldwright.wordpress.comtext/..WordPress/MU; url
4 newsnet7.wordpress.comtext/..WordPress/MU; url
4 nikolaygeorgievkotev.wordpress.comtext/..WordPress/MU; url
4 thor27.wordpress.comtext/..WordPress/MU; url
3 mannaismayaadventure.wordpress.comtext/..WordPress/MU; url
3 bibi3736.wordpress.comtext/..WordPress/MU; url
3 josseyene.wordpress.comtext/..WordPress/MU; url
3 christopherboe.wordpress.comtext/..WordPress/MU; url
167www.
81 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
46 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
35 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
3 www.text/..Google - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
157sitebot
156 www.sitebot.org/robot/text/..Mozilla/5.0 (compatible; SiteBot/0.1; url)
149toolserver
99 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
40 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
3 toolserver.org/~dispenser/text/..WebWikipedia Python/2.6 (url)
3 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/5.835
3 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
143frontpagesearch
92 frontpagesearch.nettext/..WordPress/3.1.3; url
47 frontpagesearch.netimage/..WordPress/3.1.3; url
119ac
85 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url)
22 ce.yazduni.ac.irtext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
7 www.tkl.iis.u-tokyo.ac.jp/~crawler/-Mozilla/5.0 (compatible; Steeler/3.5; url)
107sogou
98 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
3 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url)
3 www.sogou.com/docs/help/webmasters.htm#07application/vnd.php.serializedSogou web spider/4.0(url)
103wikimedia
101 tools.wikimedia.de/~daniel/text/..WikiSense (url)
103FeedBurner
102 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
101sf
33 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
33 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
32 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
98gulliway
86 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
12 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
84daum
83 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
79goo
64 help.goo.ne.jp/contact/text/..goo wikipedia (url)
11 help.goo.ne.jp/help/article/1142/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
77soso
70 help.soso.com/webspider.htmtext/..Sosospider(url)
4 help.soso.com/webspider.htm-Sosospider(url)
76semager
65 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
11 www.semager.de/blog/semager-bots/application/jsonMozilla/5.0 (compatible; Semager/1.4; url)
70sentymetr
37 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
33 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
68avantbrowser
34 www.avantbrowser.comtext/..Advanced Browser (url)
34 www.avantbrowser.comtext/..Avant Browser (url)
68scoutjet
68 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
67echonest
44 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
18 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
5 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com)
67newsgator
33 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
33 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
65feedshow
34 www.feedshow.comtext/..FeedshowOnline (url)
31 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
65jetbrains
34 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
31 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
63Anonymouse
50 Anonymouse.org/text/..url (Unix)
13 Anonymouse.org/image/..url (Unix)
59kosmix
55 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
4 www.kosmix.com/html/kosmos.htmltext/..Mozilla/5.0(compatible;Kosmos/1.0;url)
58freebase
54 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
4 www.freebase.com-metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
5180legs
41 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
9 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
51bibalex
32 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
19 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
50emining
48 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
40suggy
40 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
38apache
38 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address )
36dotnetdotcom
36 www.dotnetdotcom.org/text/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address )
36textdigger
35 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
35graemef
35 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
35tinyurl
34 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
35nguber
35 www.nguber.comtext/..g10_132_yy_x11/110510 (Mesin Pencari bahasa Indonesia; url; mail address )
34plagger
34 plagger.org/text/..Plagger/0.x.xx (url)
34rssbandit
34 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
34archive
33 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
34ponderer
34 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
34rssreader
34 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
34zootycoon
34 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
34orcabrowser
34 www.orcabrowser.comtext/..Orca Browser (url)
34hatena
31 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
3 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
33timewe
33 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
33ranchero
33 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
33winpodder
33 winpodder.comtext/..WinPodder (url)
33it-influentials
33 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
33nemui
33 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
33seebot
33 seebot.orgtext/..Lynx/2.8 (;url)
32snarfware
32 www.snarfware.com/text/..Snarfer/0.x.x (url)
32blogbridge
32 www.blogbridge.com/text/..BlogBridge 2.13 (url)
31simplepie
17 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
12 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
31kula
31 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
31feeds4all
31 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
30zipcommander
30 www.zipcommander.com/text/..1st ZipCommander (Net) - url
30flipboard
14 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
9 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
7 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
30mediawiki
30 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
26whatrhymeswith
26 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
26rcdtokyo
22 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
4 www.rcdtokyo.com/pc2m/image/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
26archive-it
17 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
9 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
25garlik
25 garlik.com/text/..GarlikCrawler/1.1 (url)
24fairshare
18 fairshare.cctext/..Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
4 fairshare.cctext/..Mozilla crawl/5.0 (compatible; fairshare.cc url)
22yioop
17 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot url)
3 yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot url)
22netnewswireapp
21 netnewswireapp.com/mac/-NetNewsWire/3.2.15 (Mac OS X; url; gzip-happy)
21spinn3r
18 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
21picsearch
18 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
20advertising
20 sl.advertising.comtext/..Mozilla/5.0 (compatible; AOL Sponsored Listing Contextual Crawler/0.8; url)
19froute
15 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
4 labs.froute.jp/pc2m/help.htmlimage/..Froute Mobile Gateway/1.0 (url)
19sourceforge
16 fess.sourceforge.jp/bot.htmltext/..Mozilla/5.0 (compatible; Fess/4.0; url)
19topsy
19 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
19alexa
19 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
19github
9 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
7 github.com/NeilCrosby/wikislurpapplication/vnd.php.serializedWikiSlurp (url)
19weblio
18 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
17ibis
11 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
5 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
17superfeedr
17 superfeedr.comapplication/xmlSuperfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
16ayna
16 www.ayna.comtext/..Mozilla/5.0 (compatible; Ayna url)
16searchtechnologies
16 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
16z-add
15 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
16rockpeaks
16 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
15turnitin
15 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
15drupal
7 drupal.org/text/..User-Agent: Drupal (url)
4 drupal.org/text/..Drupal (url)
15snap
15 www.snap.comtext/..Snapbot/1.0 (Snap Shots, url)
14cydral
9 www.cydral.comtext/..CydralSpider/3.2.6 (Cydral Image Search; url)
5 www.cydral.comimage/..CydralSpider/3.2.6 (Cydral Image Search; url)
13globalspec
13 www.globalspec.com/Ocellitext/..Ocelli/1.4 (url)
13accelobot
13 www.accelobot.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
12rootza
12 www.rootza.comapplication/xmlRootzaCrawler 2.0 (url)
12discoveryengine
12 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url)
11puritysearch
11 www.puritysearch.net/text/..Mozilla/5.0 (compatible; Purebot/1.1; url)
11wise-guys
10 www.wise-guys.nl/text/..Mozilla/4.0 (compatible; Vagabondo/4.0/CGM; url)
11linkedin
7 www.linkedin.comimage/..LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 url)
4 www.linkedin.comtext/..LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 url)
11dataparksearch
9 dataparksearch.org/bottext/..DataparkSearch/4.54-26052011 (url)
11creativecommons
11 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
10printful
7 printful.com/bot.htmltext/..Mozilla/5.0 (compatible; PrintfulBot/1.0; url)
3 printful.com/bot.htmlimage/..Mozilla/5.0 (compatible; PrintfulBot/1.0; url)
10gigablast
10 www.gigablast.com/spider.htmltext/..Gigabot/3.0 (url)
10edu
6 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url)
3 dis.sci.ntu.edu.sgtext/..momo/nutch-1.0 (momo; url; ye0001.ntu.edu.sg)
10search
10 www.search.ch/rim.htmltext/..UltraSpider3000/1.0 (url)
64,470total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
6,084PythonWikipediaBot/1.0
4,334 application/json
1,694 application/xml
56 text/..
1 -
1 image/..
868MediaWikiCrawler-Google/2.0 ( mail address )
865 text/..
3 -
662GoogleBot-Image/1.0
449 image/..
159 text/..
54 -
1 application/pdf
514php wikibot classes
499 application/vnd.php.serialized
15 text/..
477ClueBot/1.1
477 application/vnd.php.serialized
1 text/..
470Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
470 text/..
1 -
1 application/ogg
1 application/vnd.php.serialized
461GoogleBot-Image/1.0
420 text/..
21 image/..
20 application/vnd.php.serialized
1 -
407LinkParser/2.0
407 text/..
365spider
360 text/..
4 application/xml
1 application/json
1 image/..
300GoogleBot/2.1
300 text/..
1 -
1 image/..
293wikiwix-bot-3.0
289 text/..
4 image/..
1 -
239Peachy MediaWiki Bot API Version 1.0
239 application/vnd.php.serialized
1 image/..
1 text/..
234Onespot Crawler
177 application/json
53 text/..
4 -
228Answersbot
228 text/..
160 mail address
158 application/vnd.php.serialized
2 text/..
156GoogleBot-News
155 text/..
1 -
120TVersity Media Robot
120 text/..
107CorenSearchBot/1.5 en libwww-perl/5.834
107 text/..
107ClueBot/2.0
107 application/vnd.php.serialized
1 -
102MoovidaBot/0.1
102 text/..
98DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
78 text/..
18 application/xml
2 image/..
1 application/ogg
90ibo2bot
90 text/..
81SiocWikiBot
81 text/..
76Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
53 image/..
23 text/..
1 application/json
1 application/x-javascript
73Test Webbot
73 text/..
63Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
61 text/..
1 image/..
1 application/ogg
1 application/xml
1 application/vnd.php.serialized
1 audio/midi
58Pywikipediabot/2.0
58 application/json
56HTMLParser/1.6
50 text/..
6 application/json
48Opera/8.01 (J2ME/MIDP; MXit WebBot/1.3.1.0) Opera Mini/3.1
38 application/vnd.wap.xhtml+xml
5 image/..
5 text/..
1 -
45phpAPIbot 0.1
42 application/vnd.php.serialized
3 text/..
44COMODOspider/Nutch-1.0
43 text/..
1 image/..
1 application/pdf
1 -
1 video/ogg
41MLBot (www.metadatalabs.com/mlbot)
24 text/..
17 application/vnd.php.serialized
1 image/..
40DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7600.0; )
40 text/..
1 application/xml
36AnomieBOT 1.0 (TagDater)
36 application/json
36MediaWiki::Bot/3.2.6
36 application/json
36Mozilla/5.0 QunarBot/1.0
36 text/..
1 -
36Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
29 image/..
7 text/..
1 application/json
1 application/x-javascript
35GoogleBot
35 text/..
1 image/..
35SineBot/1.5.17(User:SineBot)
34 application/vnd.php.serialized
1 text/..
1 -
34Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
34 text/..
33UCMore Crawler App
33 text/..
1 -
32Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
32 text/..
1 -
32ROCKMELT-BOT
31 application/xml
1 text/..
1 -
31HTMLParser/2.0
31 text/..
1 -
1 image/..
29YBot/0.1
29 application/vnd.php.serialized
27Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.816; U; es) Presto/2.5.25 Version/10.54
21 image/..
6 text/..
1 application/x-javascript
27DotNetWikiBot/2.97 (Unix 5.10.0.0; )
27 application/xml
1 text/..
25.NET Client Parser
25 application/xml
25Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.838; U; es) Presto/2.5.25 Version/10.54
20 image/..
5 text/..
1 -
1 application/x-javascript
23COIBot/1.00
23 text/..
22AnomieBOT 1.0 (ReplaceExternalLinks2)
22 application/json
1 text/..
21DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
20 text/..
1 application/xml
20python-wikitools/1.2 (User:Mr.Z-bot)
20 application/json
20searchpark bot0.0.1
20 text/..
19gsa-crawler (Enterprise; T2-AHM48WF5YW235; mail address )
19 text/..
19VWBot - CorenSearchBot/1.5 en derivative
19 text/..
18DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
18 text/..
1 -
1 image/..
1 application/xml
18AnomieBOT 1.0 (BAGBot)
13 application/json
5 text/..
18Peachy MediaWiki Bot API Version 0.1beta
18 application/vnd.php.serialized
18DNSTallyKwBot/0.2
18 text/..
17COIBot/2.0
17 text/..
17Twitterbot/0.1
17 text/..
1 -
1 image/..
15Mozilla/4.0 (compatible; EmberSpider 0.8; Scout (a); bgft)
15 text/..
14Mozilla/5.0 (compatible; PaperLiBot/2.1)
14 text/..
1 image/..
1 application/vnd.php.serialized
14AnomieBOT 1.0 (OrphanReferenceFixer)
14 application/json
14AnomieBOT 1.0 (TemplateSubster)
14 application/json
13TrueKnowledgeBot bot mail address >
9 application/vnd.php.serialized
4 application/xml
13Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
13 text/..
13HRoestBot, de-wikipedia using pywikipedia framework
5 application/json
4 application/xml
4 text/..
13DotNetWikiBot/2.97 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
13 text/..
12Tawbot (public svn release; plwiki)
12 text/..
12Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.783; U; es) Presto/2.5.25 Version/10.54
10 image/..
2 text/..
11~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
11 text/..
10ReadonlyBot
10 text/..
10MystBot/1.5 fr libwww-perl/5.835
10 text/..
10SurakWare MediaWiki Bot/1.0
10 text/..
1 application/xml
10iGuidU wikibot 0.1 (Microsoft Windows NT 5.2.3790 Service Pack 2)
10 text/..
9Twitterbot/1.0
9 text/..
1 -
1 image/..
9('python-wikitools/1.2 (User:BernsteinBot)',)
9 application/json
9MediaWiki::Bot/3.3.1
9 application/json
9XLinkBot/1.00
9 text/..
9Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
9 text/..
1 -
1 image/..
8infraEnterprise v8 Web Crawler
8 -
1 text/..
8DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
7 text/..
1 application/xml
8FAST Enterprise Crawler 6 used by Microsoft ( mail address )
8 text/..
1 application/x-javascript
7('python-wikitools/1.2 (User:LaraBot)',)
7 application/json
6bitlybot
6 text/..
1 image/..
6DotNetWikiBot/2.91 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
4 text/..
2 application/xml
6LexxeBot/1.0 ( mail address )
6 text/..
6 mail address (Mozilla compatible)
6 text/..
1 image/..
5Handelabra WikiBot
4 application/vnd.php.serialized
1 text/..
5TheKeens bot
5 text/..
5WikiBot/1.1
5 application/vnd.php.serialized
5FAST Enterprise Crawler 6 used by viaapia (viaapia)
5 text/..
1 -
5Citation_bot; mail address
5 text/..
5Geni ircpybot 1.0
3 text/..
2 application/json
1 application/xml
5Xaldon WebSpider 2.7.b8
5 text/..
5OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
5 text/..
1 -
5DotNetWikiBot/2.9 (Unix 5.10.0.0; )
5 text/..
5OrlodrimBot/1.0
5 text/..
5FAST Enterprise Crawler 6 used by sword-group ( mail address )
5 text/..
1 -
5CheMoBot/1.00
5 text/..
4Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
4 application/xml
4Friendly Spider 1.0 contact mail address
4 text/..
4Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/24.816; en) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
4BotMapDev/1.3.516 CFNetwork/485.13.9 Darwin/10.7.0
3 image/..
1 text/..
4BotMapDev/1.3.516 CFNetwork/485.13.9 Darwin/11.0.0
3 image/..
1 text/..
1 -
4Doddebot
4 text/..
4BotMapDev/1.2.1.491 CFNetwork/485.13.9 Darwin/10.7.0
4 image/..
4DotNetWikiBot/2.96 (Unix 5.10.0.0; )
3 application/xml
1 text/..
4YourFilmsBot/0.1
4 application/json
4FAST Enterprise Crawler/5.3.4 ( mail address )
4 text/..
4Senbot/1.0
4 text/..
4Freebase Deathbot
4 text/..
4DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
4 text/..
4TextBot 0.3
4 text/..
1 -
4Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/24.838; en) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
1 -
1 application/x-javascript
4MediaWiki::Bot/3.1.6 (User:SporkBot)
4 application/json
4unblockbot/1.00
4 text/..
4gosospider "Mozilla/5.0
4 text/..
4FAST Enterprise Crawler 6 used by Swiss Re ( mail address )
4 text/..
3web corpus crawler
3 text/..
3AniBot/0.9 php/curl
3 application/vnd.php.serialized
3twinuffbot 1.0
3 text/..
3MediaWiki::Bot 3.1.5
3 application/json
3Mozilla 5.0 (Apibot 0.30b5)
3 application/vnd.php.serialized
3ReapETbot/0.2 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot>
3 text/..
3PicselSpider/1.0
3 text/..
3UCANN2_CRAWLER
3 text/..
3BotMapDev/1.3.513 CFNetwork/485.13.9 Darwin/11.0.0
2 text/..
1 image/..
3AnomieBOT 1.0 (RandomPagePicker)
3 application/json
3AnomieBOT 1.0 (AFDMergeFromCleaner)
3 application/json
3HBC Archive Indexerbot 0.9a
3 text/..
3BotMapDev/1.3.501 CFNetwork/485.13.9 Darwin/11.0.0
3 image/..
1 -
3BotMapDev/1.3.511 CFNetwork/485.13.9 Darwin/10.7.0
2 text/..
1 image/..
3Mozilla/5.0 (Bgbot 0.5)
3 text/..
3IssueCrawler
3 text/..
3AnomieBOT 1.0 (DeletionSortingCleaner)
3 application/json
3BotMapDev/1.3.501 CFNetwork/485.13.9 Darwin/10.7.0
3 image/..
14,402total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Generated on Fri, Jun 3, 2011 13:18
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.