Wikimedia Traffic Analysis Report - Crawler requests

Daily averages, based on sample period: 1 Jan 2011 - 31 Jan 2011
 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 46,869,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 410,581,000 external requests, which is 11.4%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
18,658yahoo
9,516 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
8,641 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
122 misc.yahoo.com.cn/help.htmltext/..Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
116 help.yahoo.com/help/us/ysearch/slurpapplication/x-javascriptMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
45 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
41 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! DE Slurp; url)
22 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
19 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
19 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
19 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url)
16 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp; url)
16 help.yahoo.com/help/us/ysearch/crawling/crawling-01.htmltext/..Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; url)
16 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
14 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
11 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
5 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
3 developer.yahoo.com/searchmonkey/useragentimage/..Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; url)
3 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
13,649google
9,903 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
725 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
673 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
399 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
338 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
278 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
160 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
104 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
94 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
82 code.google.com/appengineapplication/jsonPrfle.me AppEngine-Google; (url; appid: prfleme)
66 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
66 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
49 code.google.com/p/crawler4j/text/..crawler4j (url)
40 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy18)
33 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
30 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 247-0062)
30 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
23 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy4)
22 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
20 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
20 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
18 code.google.com/appenginetext/..AppEngine-Google; (url; appid: puthiyathiravidan)
18 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: image-proxy2)
18 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
18 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: mygpxy)
17 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
15 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur)
15 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ml-girgit)
14 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
13 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
12 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: proxyfly1)
12 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
10 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aadyakshar)
10 www.google.com/bot.htmlimage/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
10 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: proxyflying2)
10 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan)
9 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler01)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pa-girgit)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: te-girgit)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler02)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler00)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler04)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler03)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: suzetteklierocks)
7 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: gif-images)
7 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy19)
7 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: oohembed)
7 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
6 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; url)
5 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: proxyflying1)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nwikiproxy)
5 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mhomeroxy)
5 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: krittproxy)
4 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy18)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thakurproxy)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: twitterchitthajagat)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cmd-proxy)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gj-girgit)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mospot-test2)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
3 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
3 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy4)
3 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: flyproxy17)
3 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
3 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: proxyfly11)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
8,719facebook
5,905 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
2,443 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
291 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
37 developers.facebook.comimage/..facebookplatform/1.0 (url)
33 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
8 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
4,073bing
3,070 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
973 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
27 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
3,520google?
3,163 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
182 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
41 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
39 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
28 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
22 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
17 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
17 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
4 www.google.com/bot.htmltext/..GoogleBot/2.1/Nutch-1.1 (url; http://www.google.com/bot.html; mail address )
1,995naver
1,909 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
53 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
24 help.naver.com/delete_main.asptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
6 help.naver.com/customer_webtxt_02.jsptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
1,467yandex
1,135 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
211 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
54 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
25 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
18 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
14 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
6 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url)
1,227msn
864 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
144 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
106 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
43 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
30 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
23 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
6 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
3 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url)._
1,215baidu
653 www.baidu.jp/spider/text/..Baiduspider(url)
454 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
46 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
16 www.baidu.jp/spider/text/..DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0;url)
11 www.baidu.jp/spider/-Baiduspider(url)
9 www.baidu.com/search/spider.htm-Baiduspider(url)
8 www.baidu.jp/spider/application/xmlBaiduspider(url)
7 www.baidu.jp/spider/text/..BaiduImagespider(url)
3 www.baidu.jp/spider/text/..Baiduspider(url) ASProxy/5.5b3
512youdao
478 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
14 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
12 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
4 toolbar.youdao.com/image/..Youdao Toolbar (url)
3 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
353majestic12
349 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
300sblog
142 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
105 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
27 fulltext.sblog.cz/robot/text/..SeznamBot/2.0 (url)
22 fulltext.sblog.cz/text/..SeznamBot/3.0-alpha (url)
287traslated
287 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
276entireweb
269 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
4 www.entireweb.com/about/search_tech/speedy_spider/-Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
3 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
256exabot
242 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
13 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
227gov
226 webarchive.nlc.gov.cntext/..Mozilla/5.0 (compatible; heritrix/1.14.0 url)
225wordpress
34 arthur2rcasc.wordpress.comtext/..WordPress/MU; url
32 driwancybermuseum.wordpress.comtext/..WordPress/MU; url
21 josefboberg.wordpress.comtext/..WordPress/MU; url
16 hongkongwillie.wordpress.comtext/..WordPress/MU; url
9 thesunhillblog.wordpress.comtext/..WordPress/MU; url
8 kterrl.wordpress.comtext/..WordPress/MU; url
5 nikolayko.wordpress.comtext/..WordPress/MU; url
4 nikolaykot.wordpress.comtext/..WordPress/MU; url
4 kotenikkote.wordpress.comtext/..WordPress/MU; url
3 palashscape.wordpress.comtext/..WordPress/MU; url
3 arthur2rcasc.wordpress.comimage/..WordPress/MU; url
216scoutjet
216 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
200soso
187 help.soso.com/webspider.htmtext/..Sosospider(url)
4 help.soso.com/webspider.htm-Sosospider(url)
4 help.soso.com/soso-image-spider.htmtext/..Sosoimagespider(url)
195wikipedia
51 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.11 url
47 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.1.0 url
46 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
16 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.0.0 url
11 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.2.0 url
8 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.3.0 url
4 en.wikipedia.orgtext/..url
3 en.wikipedia.org/wiki/Wikipedia:Huggle2text/..Huggle2/2.0.1.0 url
3 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
160sogou
158 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
155php
43 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
38 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
25 pear.php.net/text/..PEAR HTTP_Request class ( url )
22 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
16 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.14
8 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.16
146yacy
17 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.26-custom; java 1.6.0_22; Europe/en) url
15 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-27-generic; java 1.6.0_20; Europe/en) url
15 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-23-generic; java 1.6.0_20; Europe/en) url
13 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) url
9 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.26; java 1.6.0_22; Europe/en) url
8 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.9-023stab052.4-smp; java 1.6.0_22; GMT/de) url
8 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.18-194.17.4.el5xen; java 1.6.0_18; Etc/en) url
7 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-4-pve; java 1.6.0_22; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_22; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.36.2; java 1.6.0_0; Europe/de) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (webportal-global; ppc Mac OS X 10.4.11; java 1.5.0_06; Europe/de) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.18-194.el5xen; java 1.6.0_23; Asia/en) url
141toolserver
103 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
22 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
8 toolserver.org/~dispenser/text/..WebWikipedia Python/2.6 (url)
4 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
131wikimedia
129 tools.wikimedia.de/~daniel/text/..WikiSense (url)
111goo
101 help.goo.ne.jp/contact/text/..goo wikipedia (url)
4 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url)
98daum
97 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
95semager
82 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
12 www.semager.de/blog/semager-bots/application/jsonMozilla/5.0 (compatible; Semager/1.4; url)
95FeedBurner
94 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
90suggy
90 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
82covario
82 www.covario.comtext/..Covario-IDS/1.0 (Covario; url; mail address )
73kosmix
64 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
9 www.kosmix.com/html/kosmos.htmltext/..Mozilla/5.0(compatible;Kosmos/1.0;url)
69www.
27 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
22 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
13 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
6 www.text/..Google - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
63ayna
63 www.ayna.comtext/..Mozilla/5.0 (compatible; Ayna url)
58sf
20 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
18 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
18 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
55emining
53 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
55freebase
54 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
54sentymetr
29 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
25 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
50grouponia
19 www.grouponia.comtext/..WordPress/3.0.4; url
14 www.grouponia.comtext/..WordPress/3.0.1; url
10 www.grouponia.comimage/..WordPress/3.0.4; url
7 www.grouponia.comimage/..WordPress/3.0.1; url
48phonifier
48 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url)
47z-add
43 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
4 w3.z-add.co.uk/linkcheck/image/..Z-Add Link Checker (url)
46dotnetdotcom
46 www.dotnetdotcom.org/text/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address )
41cogitoergosum
40 cogitoergosum.co.cctext/..WordPress/MU; url
39newsgator
19 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
18 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
39jetbrains
20 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
18 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
37textdigger
36 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
37avantbrowser
19 www.avantbrowser.comtext/..Advanced Browser (url)
18 www.avantbrowser.comtext/..Avant Browser (url)
37feedshow
19 www.feedshow.comtext/..FeedshowOnline (url)
18 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
37hatena
33 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
4 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
30ibis
15 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
12 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
3080legs
16 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
13 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
30metamoji
30 www.metamoji.com/jp/crawler.htmltext/..Mozilla/5.0 (compatible; MetamojiCrawler/1.0; url
28rcdtokyo
21 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
7 www.rcdtokyo.com/pc2m/image/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
28simplepie
17 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
8 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
28Anonymouse
16 Anonymouse.org/text/..url (Unix)
12 Anonymouse.org/image/..url (Unix)
27spinn3r
23 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
3 spinn3r.com/robot-Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
25puritysearch
25 www.puritysearch.net/text/..Mozilla/5.0 (compatible; Purebot/1.1; url)
23gigablast
23 www.gigablast.com/spider.htmltext/..Gigabot/3.0 (url)
23discoveryengine
23 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url
22microsoft
22 academic.research.microsoft.com/text/..librabot/2.0 (url)
21tinyurl
21 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
21rssreader
21 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
21digisport
21 www.digisport.rotext/..WordPress/3.0.4; url
21creativepulses
21 creativepulses.nltext/..CreativePulses Crawler (url)
21graemef
21 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
21seebot
21 seebot.orgtext/..Lynx/2.8 (;url)
20comodo
20 www.comodo.comtext/..COMODOSpider(heritrix/1.14.2 url)
20abonti
20 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
20zootycoon
20 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
20timewe
20 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
20orcabrowser
20 www.orcabrowser.comtext/..Orca Browser (url)
20dealgrater
7 dealgrater.comtext/..WordPress/3.0.4; url
6 dealgrater.comtext/..WordPress/3.0.1; url
4 dealgrater.comimage/..WordPress/3.0.1; url
3 dealgrater.comimage/..WordPress/3.0.4; url
20enwp
9 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
7 enwp.org/User:KingpinBottext/..KingpinBot (url)
3 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
20flipboard
6 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
4 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
3 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
3 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; url)
19blogbridge
19 www.blogbridge.com/text/..BlogBridge 2.13 (url)
19zipcommander
19 www.zipcommander.com/text/..1st ZipCommander (Net) - url
19turnitin
19 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
19snarfware
19 www.snarfware.com/text/..Snarfer/0.x.x (url)
19winpodder
19 winpodder.comtext/..WinPodder (url)
19rssbandit
19 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
19alexa
19 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
19ponderer
19 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
19nemui
19 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
19feeds4all
19 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
18froute
14 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
4 labs.froute.jp/pc2m/help.htmlimage/..Froute Mobile Gateway/1.0 (url)
18ranchero
18 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
18weblio
16 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
18it-influentials
18 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
17teesoft
7 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
4 www.teesoft.info/text/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
3 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 6.0; [lang code]; rv:[..]) Gecko/.. etc (url)
17topsy
17 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
17plagger
17 plagger.org/text/..Plagger/0.x.xx (url)
17kula
17 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
16mixi
9 mixi.jp/text/..mixi-mobile-converter/1.0 (url)
7 mixi.jp/image/..mixi-mobile-converter/1.0 (url)
15yioop
11 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot url)
3 yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot url)
15netnewswireapp
15 netnewswireapp.com/mac/-NetNewsWire/3.2.11 (Mac OS X; url; gzip-happy)
14fairshare
8 fairshare.cctext/..Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
5 fairshare.cctext/..Mozilla crawl/5.0 (compatible; fairshare.cc url)
14gulliway
10 www.gulliway.org/welcome.htmltext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
4 www.gulliway.org/welcome.htmlapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
14search
14 www.search.ch/rim.htmltext/..UltraSpider3000/1.0 (url)
14opensourceconnections
14 www.opensourceconnections.comtext/..Mozilla/5.0 (compatible; heritrix/2.0.2 url)
13globalspec
13 www.globalspec.com/Ocellitext/..Ocelli/1.4 (url)
13memidex
12 www.memidex.com/_bottext/..Mozilla/5.0 (compatible; Memibot/1.0; url )
13whatrhymeswith
13 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
13js-kit
13 js-kit.com/text/..JS-Kit URL Resolver, url
13rockpeaks
13 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
13gnip
13 www.gnip.com/text/..UnwindFetchor/1.0 (url)
13proximic
13 www.proximic.comtext/..Mozilla/5.0 (compatible; proximic; url)
13picsearch
11 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
12scyphus
12 draft.scyphus.co.jp/ipv6.htmltext/..IPv6'n/0.1; url
12bsurprised
11 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
12creativecommons
12 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
11cdac
10 npsf.cdac.intext/..NPSF-Nutch/Nutch-1.2 (url; mail address )
11bin-co
9 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
11archive-it
8 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
3 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
11holmes
11 holmes.getext/..HolmesBot (url)
11github
9 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
10centrum
8 morfeo.centrum.cz/bottext/..holmes/3.12.4 (url)
10wise-guys
8 www.wise-guys.nl/text/..Mozilla/4.0 (compatible; Vagabondo/4.0/CGM; url)
10linkedin
7 www.linkedin.comimage/..LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 url)
3 www.linkedin.comtext/..LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 url)
61,005total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
6,959PythonWikipediaBot/1.0
5,134 application/json
1,791 application/xml
34 text/..
1 -
1 image/..
543LinkParser/2.0
543 text/..
514php wikibot classes
346 application/vnd.php.serialized
168 text/..
1 -
497GoogleBot-Image/1.0
294 image/..
112 -
91 text/..
1 application/pdf
420Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
419 text/..
1 -
1 application/pdf
1 image/..
1 application/ogg
1 application/vnd.php.serialized
366MediaWikiCrawler-Google/1.0
366 text/..
1 -
304Onespot Crawler
236 application/json
64 text/..
4 -
288wikiwix-bot-3.0
283 text/..
4 image/..
1 -
276Answersbot
276 text/..
248ClueBot/1.1
248 application/vnd.php.serialized
1 text/..
201spider
200 text/..
1 image/..
194Mozilla/5.0 (compatible; sgbot v0.01a, mail address )
194 text/..
1 -
182gsa-crawler (Enterprise; S5-MS8QQPJ5BGWAA; mail address )
182 text/..
180Peachy MediaWiki Bot API Version 1.0
180 application/vnd.php.serialized
1 text/..
155Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
81 text/..
73 image/..
1 application/x-javascript
1 application/json
135ClueBot/2.0
135 application/vnd.php.serialized
1 -
127GoogleBot-News
126 text/..
1 -
1 image/..
126Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.8.0) Opera Mini/3.1
60 application/vnd.wap.xhtml+xml
59 image/..
7 text/..
1 -
111Peachy MediaWiki Bot API Version 0.1beta
111 application/vnd.php.serialized
1 -
87SiocWikiBot/1.0
81 application/vnd.php.serialized
6 text/..
1 -
70SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net)
49 text/..
21 image/..
69GoogleBot-Image/1.0
63 text/..
6 image/..
1 -
1 application/vnd.php.serialized
56Test Webbot
56 text/..
44AnomieBOT 1.0 (TagDater)
44 application/json
43DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7600.0; )
34 text/..
6 application/xml
3 image/..
1 application/ogg
41SineBot/1.5.17(User:SineBot)
40 application/vnd.php.serialized
1 text/..
1 -
40COMODOspider/Nutch-1.0
37 text/..
3 image/..
1 -
1 application/ogg
1 video/ogg
37EternalBot/0.2 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot>
37 text/..
35Mozilla/5.0 (compatible; suggybot v0.01a, mail address )
35 text/..
1 -
35CorenSearchBot/1.5 en libwww-perl/5.834
35 text/..
35FAST Enterprise Crawler 6 used by MS ( mail address )
35 text/..
34DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
34 text/..
1 application/xml
33crawler (xfriend server 3.0 (3.0 20100630 18:30))
33 text/..
1 -
29GoogleBot
29 text/..
1 -
1 image/..
29 mail address
28 application/vnd.php.serialized
1 text/..
1 -
29VWBot - CorenSearchBot/1.5 en derivative
29 text/..
28MLBot (www.metadatalabs.com/mlbot)
28 text/..
1 -
28HRoestBot, de-wikipedia using pywikipedia framework
13 application/xml
11 application/json
4 text/..
28TVersity Media Robot
28 text/..
27ibo2bot
27 text/..
27VWBot
27 application/json
27OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
25 text/..
2 -
25Mozilla/5.0 (compatible; 3F/ALL-PLA.NET webcrawler)
25 text/..
25AnomieBOT 1.0 (ReplaceExternalLinks2)
25 application/json
24Pywikipediabot/2.0
24 application/json
24lssbot
24 text/..
1 application/xml
23TrueKnowledgeBot bot mail address >
19 application/vnd.php.serialized
4 application/xml
23badLinks.ru`s crawler v.2
22 text/..
1 image/..
1 application/ogg
23Twitterbot/0.1
23 text/..
1 -
1 image/..
22infraEnterprise v8 Web Crawler
22 -
1 text/..
22DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; )
22 text/..
1 application/xml
1 image/..
22wikiparser/1 CFNetwork/454.11.5 Darwin/10.6.0 (x86_64) (MacPro5,1)
17 image/..
5 text/..
21HTMLParser/1.6
21 text/..
21Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
14 image/..
7 text/..
1 application/x-javascript
20BHSEOs.com Research Bot
20 text/..
1 -
19Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
19 text/..
19UCMore Crawler App
19 text/..
18Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
18 text/..
18Tawbot (public svn release; plwiki)
18 text/..
18AnomieBOT 1.0 (OrphanReferenceFixer)
18 application/json
17AnomieBOT 1.0 (BAGBot)
12 application/json
5 text/..
17Orion bot/1.0
17 text/..
1 -
16('python-wikitools/1.2 (User:BernsteinBot)',)
16 application/json
16SoxBot IRC Bot. PHP
14 application/vnd.php.serialized
2 text/..
16COIBot/1.00
16 text/..
15Dos Research Bot
15 text/..
15AnomieBOT 1.0 (TemplateSubster)
15 application/json
14DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
13 text/..
1 application/xml
14Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
14 text/..
1 image/..
13WikipediaAntiSpamBot/0.1 (hincapie.cis.upenn.edu)
13 text/..
13Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.7.0) Opera Mini/3.1
7 image/..
6 application/vnd.wap.xhtml+xml
1 text/..
12Jabse.com Crawler v.2.0 www.jabse.com/crawler.php
12 text/..
1 application/xml
12ImageSpan Crawler
12 text/..
1 image/..
1 application/ogg
12~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
12 text/..
12MystBot/1.5 fr libwww-perl/5.835
12 text/..
11HTMLParser/2.0
11 text/..
10XLinkBot/1.00
10 text/..
10AltParser/0.1
6 text/..
4 image/..
10Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot
10 text/..
1 image/..
9Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
9 text/..
9SWAT Crawler. AGH University project. In case of problem contact: mail address Thanks.
9 text/..
1 application/xml
9Handelabra WikiBot
8 text/..
1 application/vnd.php.serialized
9.NET Client Parser
9 application/xml
1 text/..
8Mozilla/5.0 QunarBot/1.0
8 text/..
8SurakWare MediaWiki Bot/1.0
8 text/..
8Mozilla/4.0 (compatible; EmberSpider 0.8; Scout (a); bgft)
8 text/..
8GunaSpider
8 text/..
8NATE.ROBOT Mozilla/5.0 (Windows; Windows NT 5.1; en-US) AppleWebKit/533.4 KHTML Chrome/5.0.375.125 Safari/533.4
8 text/..
7kmSearchBot
7 text/..
7Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/22.414; en) Presto/2.5.25 Version/10.54
4 image/..
3 text/..
7Twib::Crawler
5 text/..
2 image/..
78qiu-spider/Nutch-1.0 (this is a crawler of 8qiu; www.8qiu.com; mail address )
7 text/..
1 image/..
7Geni ircpybot 1.0
4 text/..
3 application/json
1 application/xml
7Mozilla/5.0 (X11; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3)
5 image/..
2 text/..
1 application/x-javascript
7Mozilla/5.0 (X11; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010102809 ThumbShotsBot (KFSW 3.0.6-3)
5 image/..
2 text/..
1 application/x-javascript
7('python-wikitools/1.2 (User:LaraBot)',)
7 application/json
6SiocWikiBot
6 text/..
6Tsinghua AI Lab Robot 2.0
5 text/..
1 -
6Bot/WP/EN/Alex_Bakharev/AlexNewArtBot
6 text/..
6COIBot/2.0
6 text/..
6unblockbot/1.00
6 text/..
5bitlybot
5 text/..
1 image/..
5Open Text Semantic Navigation Crawler 1.1/Nutch-1.1
4 text/..
1 -
5Mozilla/5.0 (compatible; PaperLiBot/2.1)
5 text/..
1 image/..
5DotNetWikiBot/2.94 (Microsoft Windows NT 6.1.7600.0; )
5 text/..
1 application/xml
5Freebase Deathbot
5 text/..
5AnomieBOT 1.0 (AFDMergeFromCleaner)
5 application/json
5DotNetWikiBot/2.9 (Unix 5.10.0.0; )
5 text/..
5MediaWiki::Bot/3.1.6 (User:SporkBot)
5 application/json
5Mozilla/5.0 (Bgbot 0.5)
5 text/..
5Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (iPhone; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; GoogleBot/22.414; U; en) Presto/2.5.25 Version/10.54
3 image/..
2 text/..
4DotNetWikiBot/2.92 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
4 text/..
1 application/xml
4FAST Enterprise Crawler 6 used by root ( mail address )
4 text/..
1 -
4TheKeens bot
4 text/..
4FAST Enterprise Crawler/5.3.4 ( mail address )
4 text/..
1 application/x-wiki
4betaBot
4 text/..
4AnomieBOT 1.0 (RandomPagePicker)
4 application/json
4python-wikitools/1.2 (User:Mr.Z-bot)
4 application/json
4DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
4 text/..
3Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
2 image/..
1 text/..
3MediaWiki::Bot/3.2.6
3 application/json
1 -
3PyCrawler
3 text/..
3CouponDeals.bz - Web Deals Bot
3 text/..
3Erel Bot
3 text/..
3DotNetWikiBot/2.95 (Microsoft Windows NT 5.1.2600 Service Pack 2; )
3 text/..
3GNAA-bot
3 text/..
3QBikSpider/2.0
3 text/..
1 application/opensearchdescription+xml
3HBC Archive Indexerbot 0.9a
3 text/..
3EternalBot/0.1 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot>
3 text/..
3AnomieBOT 1.0 (ReplaceExternalLinks3)
3 application/json
3DownloadSpider/5.1
2 image/..
1 text/..
3 mail address (Mozilla compatible)
3 text/..
3CheMoBot/1.00
3 text/..
3DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 2; )
2 application/xml
1 text/..
3AnomieBOT 1.0 (DeletionSortingCleaner)
3 application/json
13,673total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Generated on Tue, Feb 15, 2011 2:17
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.