Wikimedia Traffic Analysis Report - Crawler requests

Daily averages, based on sample period: 1 Dec 2010 - 31 Dec 2010
 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 45,035,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 377,435,000 external requests, which is 11.9%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
20,461yahoo
10,234 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
9,256 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
456 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
138 misc.yahoo.com.cn/help.htmltext/..Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
128 help.yahoo.com/help/us/ysearch/slurpapplication/x-javascriptMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
44 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! DE Slurp; url)
26 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
20 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp; url)
20 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
19 developer.yahoo.com/searchmonkey/useragentimage/..Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; url)
19 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url)
17 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
16 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
16 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
15 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
15 help.yahoo.com/help/us/ysearch/crawling/crawling-01.htmltext/..Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; url)
11 developer.yahoo.com/searchmonkey/useragenttext/..Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; url)
9,845google
7,027 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
680 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
610 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
273 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
201 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
186 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
111 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
89 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
67 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
63 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
49 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
39 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
37 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
32 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
29 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
24 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: image-proxy2)
23 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps)
22 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
21 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
21 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
19 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
19 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
19 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
13 www.google.com/bot.htmlimage/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
13 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: gif-images)
10 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy18)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
8 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy4)
7 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: oohembed)
7 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
6 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy)
6 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; url)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aadyakshar)
5 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
5 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
5 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: mygpxy)
4 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: chromeproxy2)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findadvise)
3 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ml-girgit)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: te-girgit)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki)
6,938facebook
4,891 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
1,754 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
263 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
20 developers.facebook.comimage/..facebookplatform/1.0 (url)
7 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
4,665google?
4,263 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
213 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
40 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
38 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
37 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
29 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
16 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
14 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
6 www.google.com/bot.htmltext/..GoogleBot/2.1/Nutch-1.1 (url; http://www.google.com/bot.html; mail address )
4,038bing
2,959 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
1,054 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
23 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
1,889naver
1,820 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
35 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
24 help.naver.com/delete_main.asptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
6 help.naver.com/customer_webtxt_02.jsptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
3 help.naver.com/robots/text/..Mozilla/5.0 (compatible; Yeti/1.0; NHN Corp.; url)
1,597yandex
1,296 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
168 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
53 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
40 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
15 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
6 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
6 yandex.com/bots-Mozilla/5.0 (compatible; YandexImages/3.0; url)
6 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url)
1,242baidu
669 www.baidu.jp/spider/text/..Baiduspider(url)
482 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
25 www.baidu.jp/spider/text/..DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0;url)
20 www.baidu.jp/spider/text/..BaiduImagespider(url)
10 www.baidu.jp/spider/-Baiduspider(url)
10 www.baidu.jp/spider/application/xmlBaiduspider(url)
9 www.baidu.com/search/spider.htm-Baiduspider(url)
9 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
4 www.baidu.jp/spider/text/..Baiduspider(url) ASProxy/5.5b3
993msn
505 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
138 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
113 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
105 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
59 search.msn.com/msnbot.htm-msnbot/2.0b (url)
40 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
22 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
6 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
548youdao
514 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
15 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
12 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
5 toolbar.youdao.com/image/..Youdao Toolbar (url)
505gov
505 webarchive.nlc.gov.cntext/..Mozilla/5.0 (compatible; heritrix/1.14.0 url)
362soso
332 help.soso.com/webspider.htmtext/..Sosospider(url)
16 help.soso.com/webspider.htm-Sosospider(url)
8 help.soso.com/soso-blog-spider.htmtext/..Sosoblogspider(url)
3 help.soso.com/soso-image-spider.htmtext/..Sosoimagespider(url)
310majestic12
300 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
4 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.2; url)
3 www.majestic12.co.uk/bot.php?image/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
274exabot
189 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
73 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
11 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
265traslated
265 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
257entireweb
251 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
4 www.entireweb.com/about/search_tech/speedy_spider/-Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
212sitebot
210 www.sitebot.org/robot/text/..Mozilla/5.0 (compatible; SiteBot/0.1; url)
196sogou
186 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
9 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url)
180scoutjet
180 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
178yacy
24 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.26; java 1.6.0_22; Europe/en) url
18 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_22; Europe/en) url
15 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-23-generic; java 1.6.0_20; Europe/en) url
14 yacy.net/bot.htmltext/..yacybot (sciencenet-global; amd64 Linux 2.6.32-26-generic; java 1.6.0_20; Europe/en) url
10 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_22; Europe/fr) url
8 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) url
8 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.18-194.26.1.el5; java 1.6.0_14; Europe/de) url
6 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.36.2; java 1.6.0_0; Europe/de) url
5 yacy.net/bot.htmltext/..yacybot (sciencenet-global; amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-23-generic; java 1.6.0_22; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-22-server; java 1.6.0_22; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.35-24-generic; java 1.6.0_20; Europe/sv) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-26-server; java 1.6.0_22; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-26-generic; java 1.6.0_22; Europe/de) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.35-23-generic; java 1.6.0_20; Europe/ru) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.26-2-686; java 1.6.0_0; America/es) url
169wordpress
21 driwancybermuseum.wordpress.comtext/..WordPress/MU; url
18 arthur2rcasc.wordpress.comtext/..WordPress/MU; url
9 nikolaykot.wordpress.comtext/..WordPress/MU; url
8 kterrl.wordpress.comtext/..WordPress/MU; url
8 josefboberg.wordpress.comtext/..WordPress/MU; url
6 klausgauger.wordpress.comtext/..WordPress/MU; url
5 jtouya.wordpress.comtext/..WordPress/MU; url
5 spacebarshift.wordpress.comtext/..WordPress/MU; url
4 mediaanakindonesia.wordpress.comtext/..WordPress/MU; url
4 hongkongwillie.wordpress.comtext/..WordPress/MU; url
3 hershlagdesign.wordpress.comtext/..WordPress/MU; url
168FeedBurner
167 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
161sblog
101 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
31 fulltext.sblog.cz/robot/text/..SeznamBot/2.0 (url)
24 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
4 fulltext.sblog.cz/text/..SeznamBot/3.0-alpha (url)
159php
49 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
38 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.14
37 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
32 pear.php.net/text/..PEAR HTTP_Request class ( url )
157wikipedia
116 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.11 url
26 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
3 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.6 url
3 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
3 ms.wikipedia.orgtext/..url
143nguber
143 www.nguber.comtext/..g10_132_yy_xxx/101213 (Mesin Pencari bahasa Indonesia; url; mail address )
133toolserver
82 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
31 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
13 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
4 toolserver.org/~dispenser/text/..WebWikipedia Python/2.6 (url)
126ayna
126 www.ayna.comtext/..Mozilla/5.0 (compatible; Ayna url)
123goo
112 help.goo.ne.jp/contact/text/..goo wikipedia (url)
7 help.goo.ne.jp/help/article/1142/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
118wikimedia
116 tools.wikimedia.de/~daniel/text/..WikiSense (url)
114kosmix
105 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
9 www.kosmix.com/html/kosmos.htmltext/..Mozilla/5.0(compatible;Kosmos/1.0;url)
106semager
89 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
17 www.semager.de/blog/semager-bots/application/jsonMozilla/5.0 (compatible; Semager/1.4; url)
89daum
88 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
66grouponia
36 www.grouponia.comtext/..WordPress/3.0.1; url
19 www.grouponia.comimage/..WordPress/3.0.1; url
7 www.grouponia.comtext/..WordPress/3.0.2; url
4 www.grouponia.comimage/..WordPress/3.0.2; url
64gulliway
41 www.gulliway.org/welcome.htmlapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
23 www.gulliway.org/welcome.htmltext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
61textdigger
61 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
56emining
54 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
56discoveryengine
55 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url
56dotnetdotcom
56 www.dotnetdotcom.org/text/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address )
56freebase
55 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
55sf
18 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
18 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
17 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
52www.
19 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
15 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
11 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
6 www.text/..Google - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
51simplepie
27 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
21 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
49ask
45 about.ask.com/en/docs/about/webmasters.shtmltext/..Mozilla/5.0 (compatible; Ask Jeeves/Teoma; url)
3 about.ask.com/en/docs/about/webmasters.shtmlimage/..Mozilla/5.0 (compatible; Ask Jeeves/Teoma; url)
46lth
46 combine.it.lth.se/text/..Combine/3 url
4480legs
33 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
9 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
39hatena
36 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
3 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
36newsgator
16 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
16 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
3 www.newsgator.com/Individuals/NetNewsWire/-NetNewsWire/3.2.8 (Mac OS X; url; gzip-happy)
36z-add
33 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
3 w3.z-add.co.uk/linkcheck/image/..Z-Add Link Checker (url)
36feedshow
18 www.feedshow.comtext/..FeedshowOnline (url)
18 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
34avantbrowser
18 www.avantbrowser.comtext/..Advanced Browser (url)
16 www.avantbrowser.comtext/..Avant Browser (url)
34jetbrains
17 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
17 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
32spinn3r
27 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
4 spinn3r.com/robot-Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
30ibis
16 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
12 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
30dealgrater
18 dealgrater.comtext/..WordPress/3.0.1; url
8 dealgrater.comimage/..WordPress/3.0.1; url
3 dealgrater.comtext/..WordPress/3.0.2; url
29seoprofiler
29 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/2.1; url )
29male
29 www.male.com.sgtext/..male.com.sg/Nutch-1.1 (url; mail address )
29sentymetr
15 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
14 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
28turnitin
27 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
25cogitoergosum
25 cogitoergosum.co.cctext/..WordPress/MU; url
24gigablast
24 www.gigablast.com/spider.htmltext/..Gigabot/3.0 (url)
23Anonymouse
13 Anonymouse.org/text/..url (Unix)
10 Anonymouse.org/image/..url (Unix)
23js-kit
23 js-kit.com/text/..JS-Kit URL Resolver, url
23enwp
14 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
9 enwp.org/User:KingpinBottext/..KingpinBot (url)
23proximic
23 www.proximic.comtext/..Mozilla/5.0 (compatible; proximic; url)
22rcdtokyo
16 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
6 www.rcdtokyo.com/pc2m/image/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
21bsurprised
20 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
20puritysearch
20 www.puritysearch.net/text/..Mozilla/5.0 (compatible; Purebot/1.1; url)
20experts-academy-review
20 experts-academy-review.com/look-what-Brendon-is-doing-nowtext/..url
20weblio
17 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
20soumission-dieu
20 www.soumission-dieu.org/bot.htmltext/..Mozilla/5.0 (compatible; sBot/1.0; url)
19tinyurl
19 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
19memidex
18 www.memidex.com/_bottext/..Mozilla/5.0 (compatible; Memibot/1.0; url )
19bin-co
15 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
4 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
19alexa
19 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
19nemui
19 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
18comodo
18 www.comodo.comtext/..COMODOSpider(heritrix/1.14.2 url)
18topsy
18 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
18winpodder
18 winpodder.comtext/..WinPodder (url)
18orcabrowser
18 www.orcabrowser.comtext/..Orca Browser (url)
18DomainUni
10 DomainUni.comtext/..Mozilla/4.0 (compatible; DomainUniBot/2.0;url)
8 DomainUni.comtext/..Mozilla/5.0(compatible;MrBoar.com/1.0;url)
18ranchero
18 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
18heartrails
6 capture.heartrails.com/text/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.9) Gecko/20100913 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.9
5 capture.heartrails.com/text/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.13) Gecko/20101213 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.13
4 capture.heartrails.com/image/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.13) Gecko/20101213 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.13
3 capture.heartrails.com/image/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.9) Gecko/20100913 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.9
18graemef
18 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
18holmes
18 holmes.getext/..HolmesBot (url)
18it-influentials
18 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
18feeds4all
18 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
17froute
13 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
4 labs.froute.jp/pc2m/help.htmlimage/..Froute Mobile Gateway/1.0 (url)
17rssreader
17 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
17zipcommander
17 www.zipcommander.com/text/..1st ZipCommander (Net) - url
17abonti
17 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
17timewe
17 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
17rssbandit
17 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
17kula
17 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
17seebot
17 seebot.orgtext/..Lynx/2.8 (;url)
16blogbridge
16 www.blogbridge.com/text/..BlogBridge 2.13 (url)
16zootycoon
16 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
16teesoft
7 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
4 www.teesoft.info/text/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
3 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 6.0; [lang code]; rv:[..]) Gecko/.. etc (url)
16snarfware
16 www.snarfware.com/text/..Snarfer/0.x.x (url)
16plagger
16 plagger.org/text/..Plagger/0.x.xx (url)
16opensourceconnections
16 www.opensourceconnections.comtext/..Mozilla/5.0 (compatible; heritrix/2.0.2 url)
16ponderer
16 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
16netnewswireapp
16 netnewswireapp.com/mac/-NetNewsWire/3.2.11 (Mac OS X; url; gzip-happy)
15flipboard
8 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; url)
4 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/1.1; url)
3 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; url)
14princexml
10 www.princexml.comtext/..Prince/7.1 (url)
4 www.princexml.comimage/..Prince/7.1 (url)
13yunrang
12 www.yunrang.com/yrspider.htmltext/..yrspider Mozilla/5.0 (compatible; YRSpider; url)
13whatrhymeswith
13 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
13bloglines
8 www.bloglines.com-Bloglines/3.1 (url; 1 subscriber)
3 www.bloglines.comtext/..Bloglines/3.1 (url; 1 subscriber)
13github
11 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
12blogscope
12 www.blogscope.net/text/..Mozilla/5.0 (compatible; BlogScope/1.0; url; U of Toronto)
12globalspec
12 www.globalspec.com/Ocellitext/..Ocelli/1.4 (url)
12wise-guys
9 www.wise-guys.nl/text/..Mozilla/4.0 (compatible; Vagabondo/4.0/CGM; url)
12accelobot
12 www.accelobot.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
12mediawiki
11 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
12search
12 www.search.ch/rim.htmltext/..UltraSpider3000/1.0 (url)
11oneriot
11 www.oneriot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 OneRiot/1.0 (url)
11mixi
6 mixi.jp/text/..mixi-mobile-converter/1.0 (url)
5 mixi.jp/image/..mixi-mobile-converter/1.0 (url)
11rockpeaks
11 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
11creativecommons
11 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
10drupal
3 drupal.org/text/..Drupal (url)
3 drupal.org/image/..Drupal (url)
58,892total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
4,417PythonWikipediaBot/1.0
3,188 application/json
1,164 application/xml
65 text/..
1 -
1 image/..
528GoogleBot-Image/1.0
250 text/..
216 image/..
62 -
1 application/pdf
1 audio/midi
472Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
472 text/..
1 -
1 application/ogg
1 application/vnd.php.serialized
448LinkParser/2.0
448 text/..
359Onespot Crawler
268 application/json
90 text/..
1 -
350php wikibot classes
345 application/vnd.php.serialized
5 text/..
1 -
275MediaWikiCrawler-Google/1.0
275 text/..
1 -
252Answersbot
252 text/..
231wikiwix-bot-3.0
227 text/..
3 image/..
1 -
221ClueBot/1.1
201 application/vnd.php.serialized
20 text/..
190Peachy MediaWiki Bot API Version 1.0
187 application/vnd.php.serialized
3 text/..
1 -
1 image/..
175spider
175 text/..
1 image/..
1 application/xml
124GoogleBot-News
124 text/..
1 -
116ClueBot/2.0
116 application/vnd.php.serialized
108gsa-crawler (Enterprise; S5-MS8QQPJ5BGWAA; mail address )
108 text/..
107Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
53 image/..
53 text/..
1 application/x-javascript
1 application/json
106Mozilla/5.0 (compatible; sgbot v0.01a, mail address )
106 text/..
96GoogleBot-Image/1.0
83 text/..
13 image/..
1 -
79Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.8.0) Opera Mini/3.1
39 image/..
36 application/vnd.wap.xhtml+xml
4 text/..
1 -
70 mail address
67 application/vnd.php.serialized
3 text/..
60Test Webbot
60 text/..
58SiocWikiBot/1.0
52 application/vnd.php.serialized
6 text/..
56CorenSearchBot/1.5 en libwww-perl/5.834
56 text/..
55COMODOspider/Nutch-1.0
51 text/..
4 image/..
1 application/ogg
54TVersity Media Robot
54 text/..
1 -
53DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7600.0; )
38 text/..
10 application/xml
5 image/..
45GoogleBot
45 text/..
42infraEnterprise v8 Web Crawler
42 -
1 text/..
40Pywikipediabot/2.0
40 application/json
38SineBot/1.5.17(User:SineBot)
37 application/vnd.php.serialized
1 text/..
378qiu-spider/Nutch-1.0 (this is a crawler of 8qiu; www.8qiu.com; mail address )
35 text/..
2 image/..
35SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net)
20 text/..
15 image/..
34SWAT Crawler. AGH University project. In case of problem contact: mail address Thanks.
34 text/..
1 application/xml
31Mozilla/5.0 (compatible; 3F/ALL-PLA.NET webcrawler)
31 text/..
30AnomieBOT 1.0 (ReplaceExternalLinks2)
30 application/json
29MLBot (www.metadatalabs.com/mlbot)
29 text/..
1 -
1 image/..
28ibo2bot
28 text/..
26crawler mail address
26 text/..
24TrueKnowledgeBot bot mail address >
20 application/vnd.php.serialized
4 application/xml
23OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
22 text/..
1 -
22VWBot - CorenSearchBot/1.5 en derivative
22 text/..
22AnomieBOT 1.0 (OrphanReferenceFixer)
22 application/json
21HRoestBot, de-wikipedia using pywikipedia framework
13 application/xml
6 application/json
2 text/..
21Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
14 image/..
7 text/..
1 application/json
1 application/x-javascript
20('python-wikitools/1.2 (User:BernsteinBot)',)
20 application/json
20wikiparser/1 CFNetwork/454.11.5 Darwin/10.5.0 (x86_64) (MacPro5,1)
15 image/..
5 text/..
20BHSEOs.com Research Bot
18 text/..
2 -
19WinkBot
19 text/..
18Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
18 text/..
1 -
18Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
18 text/..
18MystBot/1.5 fr libwww-perl/5.813
18 text/..
18SoxBot IRC Bot. PHP
16 application/vnd.php.serialized
2 text/..
17UCMore Crawler App
17 text/..
1 -
16AnomieBOT 1.0 (TemplateSubster)
16 application/json
15Tawbot (public svn release; plwiki)
15 text/..
15Twitterbot/0.1
15 text/..
1 image/..
14Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.7.0) Opera Mini/3.1
7 image/..
6 application/vnd.wap.xhtml+xml
1 text/..
14COIBot/1.00
14 text/..
13Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
13 text/..
13DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; )
13 text/..
1 application/xml
1 image/..
13Open Text Semantic Navigation Crawler 1.0/Nutch-1.1
13 text/..
1 -
13Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.6.0) Opera Mini/3.1
6 application/vnd.wap.xhtml+xml
6 image/..
1 text/..
1 -
13SuperBot/4.7.0.74 (Windows XP)
13 text/..
1 image/..
13Mozilla/5.0 (X11; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010102809 ThumbShotsBot (KFSW 3.0.6-3)
9 image/..
4 text/..
1 application/x-javascript
12DotNetWikiBot/2.72 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
12 text/..
1 application/xml
12~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
12 text/..
12TheKeens bot
12 text/..
12DotNetWikiBot/2.94 (Microsoft Windows NT 6.1.7600.0; )
12 text/..
11.NET Client Parser
11 application/xml
1 text/..
11HTMLParser/2.0
10 text/..
1 -
11COIBot/2.0
11 text/..
11Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot
11 text/..
1 image/..
11DotNetWikiBot/2.94 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
10 text/..
1 application/xml
10SurakWare MediaWiki Bot/1.0
10 text/..
1 application/xml
10Pills.com.sg/Nutch-1.1 (Pills.com.sg; mail address )
10 text/..
10DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
7 text/..
3 application/xml
9AnomieBOT 1.0 (TagDater)
9 application/json
9AnomieBOT 1.0 (BAGBot)
7 application/json
2 text/..
9VWBot
9 application/json
8Mozilla/5.0 QunarBot/1.0
8 text/..
8FAST Enterprise Crawler/5.3.4 ( mail address )
8 text/..
1 application/x-wiki
7Comodo-Certificates-Spider
7 text/..
7AnomieBOT 1.0 (WikiProjectWorker)
7 application/json
7PyCrawler
7 text/..
7XLinkBot/1.00
7 text/..
7SiocWikiBot
7 text/..
6FAST Enterprise Crawler 6 used by root ( mail address )
6 text/..
6Mozilla 5.0 (Apibot 0.30b2)
6 application/vnd.php.serialized
1 text/..
6msramlbot
6 text/..
6Geni ircpybot 1.0
3 application/json
3 text/..
6DotNetWikiBot/2.9 (Microsoft Windows NT 6.1.7600.0; )
3 application/xml
3 text/..
6NATE.ROBOT Mozilla/5.0 (Windows; Windows NT 5.1; en-US) AppleWebKit/533.4 KHTML Chrome/5.0.375.125 Safari/533.4
6 text/..
6BrianumWikicrawler at sirius.brianum.net (experimental;rev.213;Linux sirius.brianum.net 2.6.26-2-amd64 #1 SMP Wed May 12 18:03:14 UTC 2010 x86_64 GNU/Linux)
5 image/..
1 text/..
6DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 2; )
4 application/xml
2 text/..
6('python-wikitools/1.2 (User:LaraBot)',)
6 application/json
5DotNetWikiBot/2.91 (Microsoft Windows NT 6.0.6001 Service Pack 1; )
4 text/..
1 application/xml
5Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.2) Gecko Firefox/3.5 (ITABot; mail address )
3 image/..
2 text/..
1 application/json
1 application/x-javascript
5Jabse.com Crawler v.2.0 www.jabse.com/crawler.php
5 text/..
1 application/xml
5Bot/WP/EN/Alex_Bakharev/AlexNewArtBot
5 text/..
5CrowdsourcingBot/1.0
5 text/..
5Freebase Deathbot
5 text/..
5AnomieBOT 1.0 (AFDMergeFromCleaner)
5 application/json
5LNCrawl/Nutch-1.2
5 text/..
5CRAIGSPA411/Nutch-1.0 (This is a bot that will crawl PA craiglist sites. It is part of a course project for IST516 at Penn State; mail address )
5 text/..
5DotNetWikiBot/2.9 (Unix 5.10.0.0; )
5 text/..
5CheMoBot/1.00
5 text/..
4bitlybot
4 text/..
1 image/..
4WikipediaSpamBot/0.1 (hincapie.cis.upenn.edu)
4 text/..
4ImageSpan Crawler
4 text/..
4Mozilla/5.0 (compatible; PaperLiBot/2.1)
4 text/..
1 image/..
4Twib::Crawler
3 text/..
1 image/..
4Citation_bot; mail address
4 text/..
1 application/json
4AnomieBOT 1.0 (RandomPagePicker)
4 application/json
4Tsinghua AI Lab Robot
4 text/..
1 -
4python-wikitools/1.2 (User:Mr.Z-bot)
4 application/json
4Bing Research Bot
4 text/..
4Xaldon WebSpider 2.7.b8
4 text/..
1 image/..
4TOAOS SearchBot
2 image/..
2 text/..
4Mozilla/5.0 [ZSEBOT]
4 text/..
4unblockbot/1.00
4 text/..
4Mozilla/5.0 (Bgbot 0.5)
4 text/..
4AnomieBOT 1.0 (DeletionSortingCleaner)
4 application/json
3SuperBot/4.7.0.70 (Windows XP)
3 text/..
3HTMLParser/1.6
3 text/..
3Magus Bot 1.0
3 text/..
3BrianumWikicrawler at sirius.brianum.net #2 (experimental;rev.213;Linux sirius.brianum.net 2.6.26-2-amd64 #1 SMP Wed May 12 18:03:14 UTC 2010 x86_64 GNU/Linux)
3 image/..
1 text/..
3DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
3 text/..
3QBikSpider/2.0
3 text/..
1 application/xml
1 image/..
3Orion bot/1.0
3 text/..
1 -
3User:Bilderbot (derivativeFX) by Luxo on the Toolserver / PHP
3 application/vnd.php.serialized
1 text/..
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (iPhone; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; GoogleBot/22.414; U; en) Presto/2.5.25 Version/10.54
2 image/..
1 text/..
1 application/x-javascript
10,341total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Generated on Tue, Jan 11, 2011 6:34
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.