Wikimedia Traffic Analysis Report - Crawler requests

Daily averages, based on sample period: 1 Apr 2011 - 30 Apr 2011
 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 53,275,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 379,242,000 external requests, which is 14.0%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
18,764facebook
13,358 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
4,729 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
329 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
308 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
30 developers.facebook.comimage/..facebookplatform/1.0 (url)
8 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
14,027google
11,200 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
695 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
590 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
363 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
186 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
149 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
149 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
100 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
69 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
67 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
38 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
36 code.google.com/p/crawler4j/text/..crawler4j (url)
28 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
26 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
24 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
23 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
22 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
21 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
20 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
19 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
18 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
13 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; url)
11 code.google.com/appenginetext/..AppEngine-Google; (url; appid: retimeme)
11 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
10 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
9 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
8 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
7 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
7 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
7 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: oohembed)
6 code.google.com/p/ldspider/wiki/Robotstext/..ldspider (BTC 2011 crawl, mail address , url)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygpxy)
5 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findadvise)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lullar-data),gzip(gfe) (via translate.google.com)
4 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
4 sites.google.com/site/bendercrawlertext/..Mozilla/5.0 (compatible; Bender; url)
4 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
4 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
3 code.google.com/appengineapplication/jsonMozilla 3.5 AppEngine-Google; (url; appid: prfleme)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nwikiproxy)
3 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: nwikiproxy)
3 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
12,943yahoo
9,496 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
2,801 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
249 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
136 misc.yahoo.com.cn/help.htmltext/..Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
47 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
42 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! DE Slurp; url)
21 help.yahoo.com/help/us/ysearch/crawling/crawling-01.htmltext/..Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; url)
20 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-PSC/1.0 (url)
19 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
19 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url)
18 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
16 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
15 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
13 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
12 misc.yahoo.com.cn/help.html-Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
6 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
6 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
4 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible; Yahoo! Slurp; url)
5,951bing
4,473 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
1,455 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
20 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
4,563google?
4,105 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
171 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
140 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
49 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
43 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
18 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
15 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
10 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
2,052naver
1,981 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
47 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
10 help.naver.com/customer_webtxt_02.jsptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
8 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
3 help.naver.com/robots/image/..Yepi/1.0 (NHN Corp.; url)
1,783msn
1,099 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
272 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
114 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
104 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
85 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
79 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
8 search.msn.com/msnbot.htmtext/..msnbot/1.0 (url)
7 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._ (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
6 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
4 search.msn.com/msnbot.htmtext/..User-Agent :msnbot/2.0b (url)._
1,750yandex
1,412 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
189 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
67 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
35 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
20 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
10 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
8 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url)
3 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexZakladki/3.0; Dyatel; url)
1,327baidu
635 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
567 www.baidu.jp/spider/text/..Baiduspider(url)
51 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
24 www.baidu.com/search/spider.htm-Baiduspider(url)
23 www.baidu.jp/spider/text/..DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0;url)
12 www.baidu.jp/spider/text/..BaiduImagespider(url)
6 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
5 www.baidu.jp/spider/-Baiduspider(url)
388traslated
388 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
304sblog
177 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
61 fulltext.sblog.cz/text/..SeznamBot/3.0-beta (url)
29 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
27 fulltext.sblog.cz/robot/text/..SeznamBot/2.0 (url)
4 fulltext.sblog.cz/-SeznamBot/3.0-beta (url)
3 fulltext.sblog.cz/text/..SeznamBot/3.0-beta (url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
270entireweb
264 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
3 www.entireweb.com/about/search_tech/speedy_spider/-Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
265youdao
222 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
21 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
14 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
5 toolbar.youdao.com/image/..Youdao Toolbar (url)
3 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
245goso
164 www.goso.cn/spider.htmltext/..gosospider Mozilla/5.0 (compatible; GosoSpider; url)
81 www.goso.cn/aboutus.htmltext/..gosospider Mozilla/5.0 (compatible; GosoSpider; url)
228php
104 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
60 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
40 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
19 pear.php.net/text/..PEAR HTTP_Request class ( url )
225yacy
75 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-28-generic; java 1.6.0_20; Europe/en) url
35 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) url
18 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-30-generic; java 1.6.0_20; Europe/en) url
6 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-31-generic; java 1.6.0_20; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_20; Asia/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.18-194.32.1.el5.centos.plus; java 1.6.0_17; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/fr) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_24; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_24; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-31-generic; java 1.6.0_24; Europe/de) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-22-server; java 1.6.0_24; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld-global; i386 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-28-generic; java 1.6.0_24; Europe/fr) url
3 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.32-30-generic; java 1.6.0_20; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-23-server; java 1.6.0_24; Europe/en) url
214sentymetr
111 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
103 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
207exabot
117 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
79 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
8 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
3 www.exabot.com/go/robotimage/..Mozilla/5.0 (compatible; Exabot-Images/3.0; url)
203majestic12
201 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
180wikipedia
49 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.10.0 url
45 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.8.0 url
40 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
27 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.9.0 url
6 en.wikipedia.orgtext/..url
5 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.1.0 url
3 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url)
3 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
172enwp
155 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
13 enwp.org/User:KingpinBottext/..KingpinBot (url)
170wordpress
15 arthur2rcasc.wordpress.comtext/..WordPress/MU; url
12 kterrl.wordpress.comtext/..WordPress/MU; url
11 josefboberg.wordpress.comtext/..WordPress/MU; url
11 christopherboe.wordpress.comtext/..WordPress/MU; url
6 diesxdiemxdocet.wordpress.comtext/..WordPress/MU; url
6 fedupusa.wordpress.comtext/..WordPress/MU; url
5 driwancybermuseum.wordpress.comtext/..WordPress/MU; url
4 wcntransmedia.wordpress.comtext/..WordPress/MU; url
4 mannaismayaadventure.wordpress.comtext/..WordPress/MU; url
150pipl
150 www.pipl.com/bot/text/..Mozilla/5.0(compatible;PiplBot;url)
141toolserver
95 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
34 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
4 toolserver.org/~dispenser/text/..WebWikipedia Python/2.6 (url)
3 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/5.835
3 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
135sitebot
134 www.sitebot.org/robot/text/..Mozilla/5.0 (compatible; SiteBot/0.1; url)
119sogou
105 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
8 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url)
4 www.sogou.com/docs/help/webmasters.htm#07application/vnd.php.serializedSogou web spider/4.0(url)
110www.
57 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
35 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
9 www.text/..Google - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
6 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
3 www.image/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
107wikimedia
105 tools.wikimedia.de/~daniel/text/..WikiSense (url)
104echonest
61 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
28 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
10 the.echonest.com/reader/application/jsonnestReader/0.3 (discovery; url; reader at echonest.com)
5 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com)
103goo
86 help.goo.ne.jp/contact/text/..goo wikipedia (url)
10 help.goo.ne.jp/help/article/1142/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
98sf
31 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
31 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
31 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
97semager
85 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
11 www.semager.de/blog/semager-bots/application/jsonMozilla/5.0 (compatible; Semager/1.4; url)
91daum
90 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
79bibalex
51 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
28 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
72ayna
72 www.ayna.comtext/..Mozilla/5.0 (compatible; Ayna url)
71kosmix
66 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
5 www.kosmix.com/html/kosmos.htmltext/..Mozilla/5.0(compatible;Kosmos/1.0;url)
67avantbrowser
35 www.avantbrowser.comtext/..Avant Browser (url)
32 www.avantbrowser.comtext/..Advanced Browser (url)
63soso
55 help.soso.com/webspider.htmtext/..Sosospider(url)
4 help.soso.com/webspider.htm-Sosospider(url)
63gulliway
54 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
9 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
63newsgator
31 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
31 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
62feedshow
32 www.feedshow.comtext/..FeedshowOnline (url)
30 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
62jetbrains
31 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
31 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
60FeedBurner
59 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
59suggy
59 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
59discoveryengine
46 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url
8 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url)
3 discoveryengine.com/discobot.htmlimage/..Mozilla/5.0 (compatible; discobot/1.1; url
57freebase
54 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
3 www.freebase.com-metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
55bsurprised
52 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
3 bsurprised.com/text/..BSurprised WikiBox 0.1 (url)
52emining
49 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
3 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address )
5180legs
36 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
10 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
4 www.80legs.com/webcrawler.html-Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
49ac
36 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url)
10 www.clips.ua.ac.be/pages/patterntext/..Pattern/1.0 url
47Anonymouse
35 Anonymouse.org/text/..url (Unix)
12 Anonymouse.org/image/..url (Unix)
45z-add
42 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
35yunrang
24 www.yunrang.com/yrspider.htmltext/..yrspider Mozilla/5.0 (compatible; YRSpider; url)
10 www.yunrang.com/yrspider.htmltext/..gosospider Mozilla/5.0 (compatible; YRSpider; url)
35flipboard
13 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
9 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
7 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
6 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
34textdigger
33 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
34orcabrowser
34 www.orcabrowser.comtext/..Orca Browser (url)
34hatena
31 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
3 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
33timewe
33 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
33graemef
33 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
33seebot
33 seebot.orgtext/..Lynx/2.8 (;url)
32tinyurl
32 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
32rssreader
32 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
32zipcommander
32 www.zipcommander.com/text/..1st ZipCommander (Net) - url
32zootycoon
32 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
32whatrhymeswith
32 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
32it-influentials
32 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
32feeds4all
32 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
31blogbridge
31 www.blogbridge.com/text/..BlogBridge 2.13 (url)
31snarfware
31 www.snarfware.com/text/..Snarfer/0.x.x (url)
31winpodder
31 winpodder.comtext/..WinPodder (url)
31plagger
31 plagger.org/text/..Plagger/0.x.xx (url)
31ranchero
31 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
31rssbandit
31 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
31kula
31 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
31ponderer
31 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
30accelobot
30 www.accelobot.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
30simplepie
18 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
9 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
30edu
26 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url)
3 ws.nju.edu.cn/falcons/image/..Mozilla/5.0 (compatible; Falconsbot; url)
30nemui
30 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
27apache
27 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address )
26printful
14 printful.com/bot.htmltext/..Mozilla/5.0 (compatible; PrintfulBot/1.0; url)
12 printful.com/bot.htmlimage/..Mozilla/5.0 (compatible; PrintfulBot/1.0; url)
25cydral
10 www.cydral.comtext/..CydralSpider/3.2 (Cydral Image Search; url)
7 www.cydral.comimage/..CydralSpider/3.2 (Cydral Image Search; url)
5 www.cydral.comtext/..CydralSpider/3.2.6 (Cydral Image Search; url)
3 www.cydral.comimage/..CydralSpider/3.2.6 (Cydral Image Search; url)
23turnitin
23 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
22alexa
22 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
21rcdtokyo
17 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
4 www.rcdtokyo.com/pc2m/image/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
21topsy
21 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
21weblio
20 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
21spinn3r
19 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
20froute
16 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
4 labs.froute.jp/pc2m/help.htmlimage/..Froute Mobile Gateway/1.0 (url)
20bnf
12 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
8 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
20netnewswireapp
20 netnewswireapp.com/mac/-NetNewsWire/3.2.15 (Mac OS X; url; gzip-happy)
18loc
9 webarchive.loc.govtext/..Mozilla/5.0 (compatible; loc-crawler/3.0.1-SNAPSHOT-20110412.010027 url)
5 webarchive.loc.govimage/..Mozilla/5.0 (compatible; loc-crawler/3.0.1-SNAPSHOT-20110412.010027 url)
18github
9 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
6 github.com/NeilCrosby/wikislurpapplication/vnd.php.serializedWikiSlurp (url)
17sourceforge
15 fess.sourceforge.jp/bot.htmltext/..Mozilla/5.0 (compatible; Fess/4.0; url)
17ibis
11 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
4 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
15syndicat
15 www.syndicat.com/text/..Clever-BOT/2.0.2b (url)
15snap
15 www.snap.comtext/..Snapbot/1.0 (Snap Shots, url)
14search
14 www.search.ch/rim.htmltext/..UltraSpider3000/1.0 (url)
14rockpeaks
14 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
14123
14 www.123.fr/abus.htmltext/..PHP mutualise sur 123.fr - signalez les abus sur url
13gnip
13 www.gnip.com/text/..UnwindFetchor/1.0 (url)
13searchtechnologies
13 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
12umamao
12 umamao.com/text/..UmamãoBot/0.1 (url)
11globalspec
11 www.globalspec.com/Ocellitext/..Ocelli/1.4 (url)
11drupal
5 drupal.org/text/..Drupal (url)
4 drupal.org/text/..User-Agent: Drupal (url)
11holmes
11 holmes.getext/..HolmesBot (url)
11vbseo
11 www.vbseo.comtext/..Mozilla/4.0 (vBSEO; url)
11creativecommons
11 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
11picsearch
9 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
10fairshare
4 fairshare.cctext/..Mozilla crawl/5.0 (compatible; fairshare.cc url)
4 fairshare.cctext/..Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
10teesoft
5 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
10gigablast
10 www.gigablast.com/spider.htmltext/..Gigabot/3.0 (url)
10cogitoergosum
10 cogitoergosum.co.cctext/..WordPress/MU; url
10archive-it
6 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
4 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
10linkedin
7 www.linkedin.comimage/..LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 url)
3 www.linkedin.comtext/..LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 url)
10phonifier
10 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url)
70,077total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
6,919PythonWikipediaBot/1.0
4,859 application/json
2,000 application/xml
60 text/..
1 -
1 image/..
1,226GoogleBot-Image/1.0
609 text/..
516 image/..
101 -
1 application/pdf
722FAST Enterprise Crawler 6 used by Intel ( mail address )
720 text/..
2 application/xml
1 application/opensearchdescription+xml
596MediaWikiCrawler-Google/2.0 ( mail address )
582 text/..
14 -
492php wikibot classes
420 application/vnd.php.serialized
72 text/..
1 -
1 application/json
417ClueBot/1.1
416 application/vnd.php.serialized
1 text/..
417LinkParser/2.0
417 text/..
388Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
387 text/..
1 -
1 application/vnd.php.serialized
365wikiwix-bot-3.0
359 text/..
5 image/..
1 -
360spider
354 text/..
4 application/xml
2 application/json
1 image/..
359GoogleBot-Image/1.0
320 text/..
22 application/vnd.php.serialized
16 image/..
1 -
300Onespot Crawler
227 application/json
69 text/..
4 -
243Answersbot
243 text/..
235Peachy MediaWiki Bot API Version 1.0
235 application/vnd.php.serialized
1 text/..
214MediaWikiCrawler-Google/1.0
214 text/..
1 -
141TVersity Media Robot
141 text/..
1 -
1 image/..
137MoovidaBot/0.1
137 text/..
123SiocWikiBot
123 text/..
122Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
120 text/..
1 image/..
1 application/ogg
1 application/xml
1 audio/midi
108GoogleBot-News
106 text/..
2 -
98Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
73 image/..
25 text/..
1 -
1 application/json
1 application/x-javascript
86CorenSearchBot/1.5 en libwww-perl/5.834
86 text/..
83Pywikipediabot/2.0
83 application/json
81ClueBot/2.0
81 application/vnd.php.serialized
1 text/..
76 mail address
75 application/vnd.php.serialized
1 text/..
75Test Webbot
75 text/..
1 -
72ibo2bot
72 text/..
63GoogleBot/2.1
63 text/..
1 image/..
62DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
52 text/..
9 application/xml
1 image/..
54Opera/8.01 (J2ME/MIDP; MXit WebBot/1.2.0.0) Opera Mini/3.1
43 application/vnd.wap.xhtml+xml
7 image/..
4 text/..
1 -
54HTMLParser/1.6
53 text/..
1 application/json
1 image/..
1 application/vnd.php.serialized
50MLBot (www.metadatalabs.com/mlbot)
29 text/..
21 application/vnd.php.serialized
1 -
1 image/..
42Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.746; U; es) Presto/2.5.25 Version/10.54
35 image/..
7 text/..
1 application/x-javascript
38AnomieBOT 1.0 (TagDater)
38 application/json
1 text/..
38Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
31 image/..
7 text/..
1 application/json
1 application/x-javascript
37phpAPIbot 0.1
36 application/vnd.php.serialized
1 text/..
35TFR Images SpiderBot 2
35 text/..
35SineBot/1.5.17(User:SineBot)
34 application/vnd.php.serialized
1 text/..
32Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
32 text/..
1 -
31UCMore Crawler App
31 text/..
30Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
30 text/..
1 -
30MediaWiki::Bot/3.2.6
30 application/json
30python-wikitools/1.2 (User:Mr.Z-bot)
30 application/json
29GoogleBot
29 text/..
1 image/..
29badLinks.ru`s crawler v.2
28 text/..
1 image/..
1 application/x-javascript
1 application/xml
25AnomieBOT 1.0 (BAGBot)
19 application/json
6 text/..
25VWBot - CorenSearchBot/1.5 en derivative
25 text/..
23R1 Research Bot
23 text/..
1 -
23.NET Client Parser
23 application/xml
1 text/..
23OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
21 text/..
2 -
22Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.783; U; es) Presto/2.5.25 Version/10.54
17 image/..
5 text/..
22Mozilla/4.0 (compatible; EmberSpider 0.8; Scout (a); bgft)
22 text/..
22FAST Enterprise Crawler 6 used by ESP ( mail address )
22 text/..
21Mozilla/5.0 QunarBot/1.0
21 text/..
1 -
21AnomieBOT 1.0 (ReplaceExternalLinks2)
21 application/json
20DotNetWikiBot/2.97 (Unix 5.10.0.0; )
20 application/xml
1 text/..
19DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
19 text/..
1 application/xml
18DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7600.0; )
18 text/..
1 application/xml
17Peachy MediaWiki Bot API Version 0.1beta
17 application/vnd.php.serialized
17AnomieBOT 1.0 (OrphanReferenceFixer)
17 application/json
16Twitterbot/0.1
16 text/..
1 -
1 image/..
1 application/ogg
14HRoestBot, de-wikipedia using pywikipedia framework
6 application/json
4 application/xml
4 text/..
14yolinkBot
14 text/..
14AnomieBOT 1.0 (TemplateSubster)
14 application/json
13Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.760; U; es) Presto/2.5.25 Version/10.54
10 image/..
3 text/..
1 application/x-javascript
13DNSTally.com Bot
13 text/..
12Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
12 text/..
1 -
1 image/..
11TFR Images SpiderBot 4
11 text/..
11SurakWare MediaWiki Bot/1.0
11 text/..
11Tawbot (public svn release; plwiki)
11 text/..
11COMODOspider/Nutch-1.0
10 text/..
1 image/..
10~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
10 text/..
10Mozilla/5.0 (compatible; PaperLiBot/2.1)
10 text/..
1 image/..
10ReapETbot/0.2 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot>
10 text/..
10YBot/0.1
10 application/vnd.php.serialized
10SearchBot
10 text/..
9('python-wikitools/1.2 (User:BernsteinBot)',)
9 application/json
9ReadonlyBot
9 text/..
9TrueKnowledgeBot bot mail address >
5 application/vnd.php.serialized
4 application/xml
9DotNetWikiBot/2.94 (Microsoft Windows NT 6.1.7600.0; )
9 text/..
9COIBot/2.0
9 text/..
9TheSEOBay Spider
9 text/..
9COIBot/1.00
9 text/..
8infraEnterprise v8 Web Crawler
8 -
1 text/..
8FAST Enterprise Crawler/6.7.8 ( mail address )
8 text/..
1 -
8TheKeens bot
8 text/..
8HTMLParser/2.0
8 text/..
7lssbot
7 text/..
7DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
5 text/..
2 application/xml
7Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/24.746; en) Presto/2.5.25 Version/10.54
5 image/..
2 text/..
7XLinkBot/1.00
7 text/..
7Minisearch_Spider
7 text/..
1 -
7 mail address (Mozilla compatible)
7 text/..
1 image/..
7MSR-ISRCCrawler
7 text/..
6Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
6 application/xml
1 text/..
6Handelabra WikiBot
5 application/vnd.php.serialized
1 text/..
6cis455crawler
6 text/..
1 application/rsd+xml
1 application/opensearchdescription+xml
1 image/..
6DotNetWikiBot/2.96 (Unix 5.10.0.0; )
4 application/xml
2 text/..
6User-Agent: MyWikiBot/0.2
6 image/..
6Geni ircpybot 1.0
3 application/json
3 text/..
6DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
5 text/..
1 application/xml
6NATE.ROBOT Mozilla/5.0 (Windows; Windows NT 5.1; en-US) AppleWebKit/533.4 KHTML Chrome/5.0.375.125 Safari/533.4
6 text/..
6('python-wikitools/1.2 (User:LaraBot)',)
6 application/json
5bitlybot
5 text/..
1 image/..
5Opera/9.80 (J2ME/MIDP; MXit WebBot 1.3.0.0; en) Presto/2.4.15
5 text/..
1 image/..
5Doddebot
5 text/..
5Freebase Deathbot
5 text/..
5AnomieBOT 1.0 (AFDMergeFromCleaner)
5 application/json
5DotNetWikiBot/2.9 (Unix 5.10.0.0; )
5 text/..
5DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
5 text/..
1 application/xml
4DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 2; )
4 text/..
4FAST Enterprise Crawler/5.3.4 ( mail address )
4 text/..
4FAST Enterprise Crawler 6 used by viaapia (viaapia)
4 text/..
4DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
3 text/..
1 application/xml
4percolateSPIDERobot
4 text/..
1 -
1 image/..
1 application/xml
4DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
4 text/..
4Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/24.783; en) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
4HBC Archive Indexerbot 0.9a
4 text/..
4wikiparser/1 CFNetwork/454.11.12 Darwin/10.7.0 (x86_64) (MacPro5,1)
3 image/..
1 text/..
4TextBot 0.3
4 text/..
1 -
4MediaWiki::Bot/3.1.6 (User:SporkBot)
4 application/json
4Jyxobot/1
4 text/..
1 application/xml
4Mozilla/5.0 (Bgbot 0.5)
4 text/..
3Twitterbot/1.0
3 text/..
1 image/..
3Articles Crawler Bot
3 text/..
3AniBot/0.9 php/curl
3 application/vnd.php.serialized
1 image/..
3Friendly Spider 1.0 contact mail address
3 text/..
3Moholibot
3 text/..
3MystBot/1.5 fr libwww-perl/5.835
3 text/..
3Jbot
3 text/..
3Citation_bot; mail address
3 text/..
3AnomieBOT 1.0 (RandomPagePicker)
3 application/json
3Senbot/1.0
3 text/..
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (iPhone; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; GoogleBot/24.783; U; en) Presto/2.5.25 Version/10.54
2 image/..
1 text/..
3Nutch 1.2/Nutch-1.2 (Facet Engine Nutch Crawler; mail address )
3 text/..
1 application/ogg
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (iPhone; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; GoogleBot/24.746; U; en) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
3Web History Research Spider
3 text/..
3WikiBot/0.1
3 text/..
3AlertiaBot/1.1
3 text/..
3unblockbot/1.00
3 text/..
3IssueCrawler
3 text/..
16,023total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Generated on Sun, May 8, 2011 0:28
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.