Wikimedia Traffic Analysis Report - Crawler requests

Daily averages, based on sample period: 1 Nov 2010 - 30 Nov 2010
 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 41,684,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 379,902,000 external requests, which is 11.0%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
13,722yahoo
6,938 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
6,391 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
95 misc.yahoo.com.cn/help.htmltext/..Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
53 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
46 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
41 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! DE Slurp; url)
23 developer.yahoo.com/searchmonkey/useragentimage/..Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; url)
18 developer.yahoo.com/searchmonkey/useragenttext/..Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; url)
14 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
14 help.yahoo.com/help/us/ysearch/crawling/crawling-01.htmltext/..Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; url)
14 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url)
13 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp; url)
13 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
12 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
10 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
7 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
4 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
4 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
4 help.yahoo.com/help/us/ysearch/slurpapplication/x-javascriptMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
7,948google
5,324 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
774 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
284 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
244 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
222 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
190 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
128 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
108 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
67 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
66 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
57 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
51 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
45 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
31 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
25 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: image-proxy2)
24 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
23 code.google.com/appenginetext/..AppEngine-Google; (url; appid: npiv82)
22 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: chromeproxy2)
21 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
21 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
20 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findadvise)
18 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: gif-images)
17 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
15 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
14 www.google.com/bot.htmlimage/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
13 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: chromeproxy1000)
13 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
11 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
8 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; url)
8 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: oohembed)
7 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: jpg-images)
7 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
6 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
6 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: allinone)
4 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.1.14) Gecko/20080404 (FoxPlus) Firefox/2.0.0.14 AppEngine-Google; (url; appid: 1000hottest)
3 code.google.com/p/crawler4j/text/..crawler4j (url)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lullar-data),gzip(gfe) (via translate.google.com)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
3 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
7,739facebook
5,148 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
2,182 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
287 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
100 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
12 developers.facebook.comimage/..facebookplatform/1.0 (url)
9 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
3,534bing
2,694 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
808 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
30 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
3,027google?
2,716 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
150 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
40 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
30 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
26 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
22 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
14 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
11 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
4 www.google.com/bot.htmlapplication/xmlSAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
4 www.google.com/bot.htmlapplication/xmlDoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
1,795naver
1,722 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
32 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
30 help.naver.com/delete_main.asptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
9 help.naver.com/customer_webtxt_02.jsptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
1,450yandex
1,173 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
136 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
66 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
38 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
15 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
6 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
6 yandex.com/bots-Mozilla/5.0 (compatible; YandexImages/3.0; url)
6 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url)
1,041baidu
529 www.baidu.jp/spider/text/..Baiduspider(url)
460 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
24 www.baidu.jp/spider/text/..DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0;url)
10 www.baidu.jp/spider/application/xmlBaiduspider(url)
6 www.baidu.jp/spider/-Baiduspider(url)
5 www.baidu.com/search/spider.htm-Baiduspider(url)
3 www.baidu.jp/spider/text/..BaiduImagespider(url)
993msn
352 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
244 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
122 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
101 search.msn.com/msnbot.htm-msnbot/2.0b (url)
94 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
45 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
14 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
8 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
4 search.msn.com/msnbot.htmtext/..msnbot/1.0 (url)
3 search.msn.com/msnbot.htmapplication/xmlmsnbot/2.0b (url)._
587ask
479 about.ask.com/en/docs/about/webmasters.shtmltext/..Mozilla/5.0 (compatible; Ask Jeeves/Teoma; url)
104 about.ask.com/en/docs/about/webmasters.shtml-Mozilla/5.0 (compatible; Ask Jeeves/Teoma; url)
554youdao
482 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
33 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
13 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
11 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
10 www.youdao.com/help/webmaster/spider/image/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
5 toolbar.youdao.com/image/..Youdao Toolbar (url)
483majestic12
458 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
11 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.2; url)
8 www.majestic12.co.uk/bot.php?text/..User-Agent :Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
409sitebot
407 www.sitebot.org/robot/text/..Mozilla/5.0 (compatible; SiteBot/0.1; url)
400exabot
216 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
170 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
11 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
337soso
315 help.soso.com/webspider.htmtext/..Sosospider(url)
12 help.soso.com/webspider.htm-Sosospider(url)
8 help.soso.com/soso-blog-spider.htmtext/..Sosoblogspider(url)
335scoutjet
335 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
289entireweb
284 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
4 www.entireweb.com/about/search_tech/speedy_spider/-Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
267traslated
267 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
245sogou
233 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
11 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url)
202wordpress
27 josefboberg.wordpress.comtext/..WordPress/MU; url
26 driwancybermuseum.wordpress.comtext/..WordPress/MU; url
19 arthur2rcasc.wordpress.comtext/..WordPress/MU; url
7 mannaismayaadventure.wordpress.comtext/..WordPress/MU; url
6 musiquefreak.wordpress.comtext/..WordPress/MU; url
5 spacebarshift.wordpress.comtext/..WordPress/MU; url
4 klausgauger.wordpress.comtext/..WordPress/MU; url
4 benabb.wordpress.comtext/..WordPress/MU; url
3 todealornot.wordpress.comtext/..WordPress/MU; url
3 kterrl.wordpress.comtext/..WordPress/MU; url
3 terrytao.wordpress.comtext/..WordPress/MU; url
201ayna
201 www.ayna.comtext/..Mozilla/5.0 (compatible; Ayna url)
194yacy
25 yacy.net/bot.htmltext/..yacybot (sciencenet-global; amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) url
15 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-23-generic; java 1.6.0_22; Europe/en) url
14 yacy.net/bot.htmltext/..yacybot (sciencenet-global; amd64 Linux 2.6.32-25-generic; java 1.6.0_18; Europe/en) url
10 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_22; Europe/fr) url
10 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-3-pve; java 1.6.0_18; Europe/cs) url
9 yacy.net/bot.htmltext/..yacybot (amd64 FreeBSD 8.0-RELEASE-p3; java 1.6.0_07; Europe/en) url
6 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-26-generic; java 1.6.0_22; Europe/de) url
6 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.28.3-vs2.3.0.36.4; java 1.6.0_20; Universal/en) url
5 yacy.net/bot.htmltext/..yacybot (amd64 Linux 2.6.32-25-generic; java 1.6.0_18; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Vista 6.0; java 1.6.0_23-ea; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (amd64 Linux 2.6.18-194.17.4.el5; java 1.6.0_17; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/de) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-25-generic; java 1.6.0_22; Europe/de) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.26-2-686; java 1.6.0_0; America/es) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_20; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.18-194.26.1.el5; java 1.6.0_14; Europe/de) url
4 yacy.net/bot.htmltext/..yacybot (i386 Linux 2.6.35-22-generic-pae; java 1.6.0_20; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_23-ea; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-25-generic; java 1.6.0_18; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-26-generic; java 1.6.0_18; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (amd64 Linux 2.6.32-22-server; java 1.6.0_18; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.24-1-686; java 1.6.0_0; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-23-server; java 1.6.0_20; Europe/en) url
178wikipedia
118 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.11 url
24 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
9 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.9 url
7 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.6 url
7 en.wikipedia.orgtext/..url
5 fr.wikipedia.org/wiki/Utilisateur:geobotapplication/jsongeobot, see url (uses Perl MediaWiki::API)
4 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
146grouponia
98 www.grouponia.comtext/..WordPress/3.0.1; url
48 www.grouponia.comimage/..WordPress/3.0.1; url
143sblog
71 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
27 fulltext.sblog.cz/robot/text/..SeznamBot/2.0 (url)
22 fulltext.sblog.cz/text/..SeznamBot/3.0-alpha (url)
21 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
143FeedBurner
142 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
141php
46 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
35 pear.php.net/text/..PEAR HTTP_Request class ( url )
29 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.14
29 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
140toolserver
93 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
37 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
3 toolserver.org/~dispenser/text/..WebWikipedia Python/2.6 (url)
3 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/5.835
129kosmix
117 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
12 www.kosmix.com/html/kosmos.htmltext/..Mozilla/5.0(compatible;Kosmos/1.0;url)
126princexml
74 www.princexml.comtext/..Prince/7.1 (url)
41 www.princexml.comimage/..Prince/7.1 (url)
6 www.princexml.comimage/..Prince/7.2 (url)
5 www.princexml.comtext/..Prince/7.2 (url)
123wikimedia
120 tools.wikimedia.de/~daniel/text/..WikiSense (url)
108goo
101 help.goo.ne.jp/contact/text/..goo wikipedia (url)
3 help.goo.ne.jp/help/article/1142/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
103daum
103 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
90semager
65 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
25 www.semager.de/blog/semager-bots/application/jsonMozilla/5.0 (compatible; Semager/1.4; url)
72chug
72 crawler.chug.nettext/..Mozilla/5.0 (compatible; heritrix/3.0.0 url)
64textdigger
64 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
64sf
23 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
20 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
19 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
62dotnetdotcom
62 www.dotnetdotcom.org/text/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address )
58emining
56 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
57freebase
57 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
56www.
23 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
12 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
11 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
8 www.text/..Google - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
54gov
53 webarchive.nlc.gov.cntext/..Mozilla/5.0 (compatible; heritrix/1.14.0 url)
5480legs
51 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
49newsgator
22 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
22 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
3 www.newsgator.com/Individuals/NetNewsWire/-NetNewsWire/3.2.8 (Mac OS X; url; gzip-happy)
45archive-it
30 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox
15 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox
44cogitoergosum
43 cogitoergosum.co.cctext/..WordPress/MU; url
44jetbrains
22 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
22 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
43avantbrowser
22 www.avantbrowser.comtext/..Advanced Browser (url)
21 www.avantbrowser.comtext/..Avant Browser (url)
42feedshow
21 www.feedshow.comtext/..FeedshowOnline (url)
21 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
39hatena
35 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
4 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
34gulliway
27 www.gulliway.org/welcome.htmlapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
7 www.gulliway.org/welcome.htmltext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
34spinn3r
28 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
4 spinn3r.com/robot-Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
30weblio
28 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
28heartrails
16 capture.heartrails.com/text/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.9) Gecko/20100913 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.9
12 capture.heartrails.com/image/..Mozilla/5.0 (X11; Linux i686; en-US; rv:1.9.2.9) Gecko/20100913 HeartRails_Capture/1.0.4 (url) Namoroka/3.6.9
27flaptor
27 www.flaptor.com/text/..HounderCrawl/Nutch-0.9 (Hounder Search Bot; url)
27simplepie
15 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
9 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
26abonti
26 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
26discoveryengine
23 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url
24zootycoon
24 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
24timewe
24 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
24ibis
14 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
10 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
23tinyurl
23 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
23blogbridge
23 www.blogbridge.com/text/..BlogBridge 2.13 (url)
23snarfware
23 www.snarfware.com/text/..Snarfer/0.x.x (url)
23rssbandit
23 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
23graemef
23 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
23it-influentials
23 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
23nemui
23 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
22rssreader
22 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
22zipcommander
22 www.zipcommander.com/text/..1st ZipCommander (Net) - url
22rcdtokyo
16 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
6 www.rcdtokyo.com/pc2m/image/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
22orcabrowser
22 www.orcabrowser.comtext/..Orca Browser (url)
22plagger
22 plagger.org/text/..Plagger/0.x.xx (url)
22ranchero
22 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
22kula
22 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
22ponderer
22 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
22seebot
22 seebot.orgtext/..Lynx/2.8 (;url)
21teesoft
8 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
4 www.teesoft.info/text/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
3 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 6.0; [lang code]; rv:[..]) Gecko/.. etc (url)
21winpodder
21 winpodder.comtext/..WinPodder (url)
21innovationtune
21 www.innovationtune.comtext/..innoBot/0.2 (url)
21kalooga
13 www.kalooga.com/info.html?page=crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url)
8 www.kalooga.com/info.html?page=crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url)
21yioop
20 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot url)
21bsurprised
19 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
20puritysearch
20 www.puritysearch.net/text/..Mozilla/5.0 (compatible; Purebot/1.1; url)
20gigablast
20 www.gigablast.com/spider.htmltext/..Gigabot/3.0 (url)
20Anonymouse
11 Anonymouse.org/text/..url (Unix)
9 Anonymouse.org/image/..url (Unix)
20yumroo
20 www.yumroo.aztext/..YUMROO/yumroo.search.bot-1.0 (search bot of url; mail address )
20feeds4all
20 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
19kindsight
19 www.kindsight.net/en/kscrawlertext/..KSCrawler/Nutch-1.0 (url; mail address )
19holmes
19 holmes.getext/..HolmesBot (url)
18seoprofiler
18 www.seoprofiler.com/bottext/..Mozilla/5.0 (compatible; spbot/2.1; url )
18froute
14 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
4 labs.froute.jp/pc2m/help.htmlimage/..Froute Mobile Gateway/1.0 (url)
18z-add
17 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
18flipboard
8 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; url)
5 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; url)
5 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/1.1; url)
17comodo
17 www.comodo.comtext/..COMODOSpider(url)
17oneriot
17 www.oneriot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 OneRiot/1.0 (url)
17alexa
17 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
17js-kit
17 js-kit.com/text/..JS-Kit URL Resolver, url
16topsy
16 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
16bin-co
13 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
3 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
16opensourceconnections
16 www.opensourceconnections.comtext/..Mozilla/5.0 (compatible; heritrix/2.0.2 url)
15mixi
8 mixi.jp/text/..mixi-mobile-converter/1.0 (url)
7 mixi.jp/image/..mixi-mobile-converter/1.0 (url)
14blx
14 www.blx.pl/crawlertext/..Mozilla/5.0 (compatible; heritrix/2.0.2 url)
14github
12 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
13wise-guys
9 www.wise-guys.nl/text/..Mozilla/4.0 (compatible; Vagabondo/4.0/CGM; url)
12globalspec
12 www.globalspec.com/Ocellitext/..Ocelli/1.4 (url)
12bloglines
7 www.bloglines.com-Bloglines/3.1 (url; 1 subscriber)
3 www.bloglines.comtext/..Bloglines/3.1 (url; 1 subscriber)
12metamoji
12 www.metamoji.com/jp/crawler.htmltext/..Mozilla/5.0 (compatible; MetamojiCrawler/1.0; url
12picsearch
10 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
11nguber
11 www.nguber.comtext/..g10_132_yy_xxx/101105 (Mesin Pencari bahasa Indonesia; url; mail address )
11chitanka
11 chitanka.infotext/..Mylib (url)
11edu
3 it.usyd.edu.au/~smerity/schwa/crawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
3 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url)
11mnemoo
11 www.mnemoo.com/about/spidertext/..Mnemoo Intelligent Spider/0.1alpha (compatible; See url)
11netarkivet
9 netarkivet.dk/website/info.htmltext/..Mozilla/5.0 (compatible; heritrix/1.5.0-200506132127 url)
11rockpeaks
11 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
11netnewswireapp
11 netnewswireapp.com/mac/-NetNewsWire/3.2.11 (Mac OS X; url; gzip-happy)
10creativecommons
10 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
49,484total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
3,902PythonWikipediaBot/1.0
2,797 application/json
985 application/xml
120 text/..
1 -
1 image/..
515ClueBot/1.1
334 application/vnd.php.serialized
181 text/..
486LinkParser/2.0
486 text/..
467Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
467 text/..
1 -
1 application/vnd.php.serialized
461GoogleBot-Image/1.0
195 text/..
170 image/..
96 -
1 application/pdf
292php wikibot classes
282 application/vnd.php.serialized
10 text/..
1 -
277Answersbot
277 text/..
261Onespot Crawler
194 application/json
67 text/..
223Peachy MediaWiki Bot API Version 1.0
218 application/vnd.php.serialized
5 text/..
1 -
201wikiwix-bot-3.0
176 text/..
25 image/..
1 -
162spider
162 text/..
1 -
1 image/..
136GoogleBot-Image/1.0
122 text/..
14 image/..
1 -
118plantspedia data crawler
118 text/..
115Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
59 image/..
55 text/..
1 application/x-javascript
1 application/json
110taxobot; mail address
110 text/..
1 application/json
107ClueBot/2.0
107 application/vnd.php.serialized
1 -
106GoogleBot-News
106 text/..
1 -
103TheKeens bot
103 text/..
78crawler mail address
78 text/..
62AnomieBOT 1.0 (ReplaceExternalLinks2)
62 application/json
1 text/..
55Test Webbot
55 text/..
49Mozilla/5.0 (compatible; sgbot v0.01a, mail address )
49 text/..
46lssbot
46 text/..
1 application/xml
44Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.6.0) Opera Mini/3.1
22 application/vnd.wap.xhtml+xml
20 image/..
2 text/..
1 -
44SiocWikiBot/1.0
40 application/vnd.php.serialized
4 text/..
41CorenSearchBot/1.5 en libwww-perl/5.834
41 text/..
40COIBot/2.0
40 text/..
39Pywikipediabot/2.0
39 application/json
39DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7600.0; )
29 text/..
7 application/xml
3 image/..
1 application/ogg
1 audio/midi
38ibo2bot
38 text/..
38SineBot/1.5.17(User:SineBot)
37 application/vnd.php.serialized
1 text/..
1 -
31GoogleBot
31 text/..
1 image/..
28MLBot (www.metadatalabs.com/mlbot)
28 text/..
1 application/xml
28COMODOspider/Nutch-1.0
25 text/..
3 image/..
1 application/pdf
1 -
1 application/ogg
28MSR WolframAlpha Bot
22 text/..
6 image/..
27infraEnterprise v8 Web Crawler
27 -
1 text/..
27DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; )
27 text/..
1 application/xml
24Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
16 image/..
8 text/..
1 application/json
1 application/x-javascript
248qiu-spider/Nutch-1.0 (this is a crawler of 8qiu; www.8qiu.com; mail address )
23 text/..
1 image/..
24dicbot 1.0
24 text/..
23HRoestBot, de-wikipedia using pywikipedia framework
18 application/xml
3 application/json
2 text/..
22Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
22 text/..
1 -
22Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
22 text/..
22UCMore Crawler App
22 text/..
22AnomieBOT 1.0 (OrphanReferenceFixer)
22 application/json
21VWBot - CorenSearchBot/1.5 en derivative
21 text/..
20CrowdsourcingBot/1.0
20 text/..
19BHSEOs.com Research Bot
19 text/..
1 -
19UniFind Site Spider; email mail address
19 text/..
1 -
18HTMLParser/2.0
17 text/..
1 -
18 mail address (Mozilla compatible)
18 text/..
17('python-wikitools/1.2 (User:BernsteinBot)',)
17 application/json
17MoovidaBot/0.1
17 text/..
16AnomieBOT 1.0 (TemplateSubster)
16 application/json
15Mozilla/5.0 (compatible; 3F/ALL-PLA.NET webcrawler)
15 text/..
15Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.5.0) Opera Mini/3.1
8 application/vnd.wap.xhtml+xml
6 image/..
1 text/..
15HLTC-HKUST Research Bot 0.1 - E. Prochasson
10 application/json
5 text/..
15COIBot/1.00
15 text/..
13HTMLParser/1.6
13 text/..
13DotNetWikiBot/2.94 (Microsoft Windows NT 6.1.7600.0; )
13 text/..
12SWAT Crawler. AGH University project. In case of problem contact: mail address Thanks.
12 text/..
1 -
1 application/xml
12Tawbot (public svn release; plwiki)
12 text/..
12Mozilla 5.0 (Apibot 0.20b1)
12 application/vnd.php.serialized
11~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
11 text/..
11Bot/WP/EN/Daniel/MediationBot1/1.2
11 text/..
11gsa-crawler (Enterprise; T1-E5BRH8AVBGWJT; mail address )
11 text/..
11Twitterbot/0.1
11 text/..
1 image/..
1 application/ogg
10TrueKnowledgeBot bot mail address >
7 application/vnd.php.serialized
3 application/xml
10DotNetWikiBot/2.72 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
10 text/..
1 application/xml
10DotNetWikiBot/2.95 (Microsoft Windows NT 5.1.2600 Service Pack 2; )
10 text/..
10TVersity Media Robot
10 text/..
10Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot
10 text/..
1 image/..
9SexShop.sg/Nutch-1.1 (www.SexShop.sg; mail address )
9 text/..
9.NET Client Parser
9 application/xml
1 text/..
9SurakWare MediaWiki Bot/1.0
9 text/..
1 application/xml
8 mail address
7 application/vnd.php.serialized
1 text/..
8MystBot/1.5 fr libwww-perl/5.836
8 text/..
8YBot/0.1
8 application/vnd.php.serialized
7XLinkBot/1.00
7 text/..
7Geni ircpybot 1.0
4 text/..
3 application/json
1 application/xml
7MSR-ISRCCrawler
7 text/..
1 image/..
7DotNetWikiBot/2.94 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
6 text/..
1 application/xml
1 image/..
6bitlybot
6 text/..
1 image/..
6FeedlyBot
6 text/..
6wikiparser/1 CFNetwork/454.11.5 Darwin/10.5.0 (x86_64) (MacPro5,1)
5 image/..
1 text/..
6oogles.com Research Bot
6 text/..
6AnomieBOT 1.0 (SourceUploader)
6 application/json
6Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )"
6 application/xml
1 text/..
6NATE.ROBOT Mozilla/5.0 (Windows; Windows NT 5.1; en-US) AppleWebKit/533.4 KHTML Chrome/5.0.375.125 Safari/533.4
6 text/..
6Mozilla/5.0 (X11; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010102809 ThumbShotsBot (KFSW 3.0.6-3)
4 image/..
2 text/..
1 application/x-javascript
6('python-wikitools/1.2 (User:LaraBot)',)
6 application/json
5'zhiqi_nutch spider'/Nutch-1.2
5 text/..
1 -
5Jabse.com Crawler v.2.0 www.jabse.com/crawler.php
5 text/..
5WinkBot
5 text/..
5SiocWikiBot
5 text/..
5Twib::Crawler
4 text/..
1 image/..
5betaBot
5 text/..
5OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
5 text/..
5MediaWiki::Bot/3.1.6 (User:SporkBot)
5 application/json
5CheMoBot/1.00
5 text/..
5Mozilla/5.0 (Bgbot 0.5)
5 text/..
4Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
4 text/..
4easycotes bot
4 text/..
4Mozilla/5.0 QunarBot/1.0
4 text/..
4FAST Enterprise Crawler/5.3.4 ( mail address )
4 text/..
1 application/x-wiki
4Bot/WP/EN/Alex_Bakharev/AlexNewArtBot
4 text/..
4DotNetWikiBot/2.95 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
4 text/..
4AnomieBOT 1.0 (RandomPagePicker)
4 application/json
4Freebase Deathbot
4 text/..
4AnomieBOT 1.0 (AFDMergeFromCleaner)
4 application/json
4unblockbot/1.00
4 text/..
4IssueCrawler
4 text/..
3Mozilla/5.0 (iPhone; CPU iPhone OS 4_0_1 like Mac OS X; fr-fr) OrangeBot-Mobile AppleWebKit/532.9 (KHTML
3 image/..
1 text/..
3Magus Bot 1.0
3 text/..
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (iPhone; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; GoogleBot/21.549; U; en) Presto/2.5.25 Version/10.54
2 image/..
1 text/..
3GNAA-bot
3 text/..
3DotNetWikiBot/2.9 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
3 text/..
3python-wikitools/1.2 (User:Mr.Z-bot)
3 application/json
3DotNetWikiBot/2.9 (Microsoft Windows NT 6.1.7600.0; )
2 text/..
1 application/xml
3HBC Archive Indexerbot 0.9a
3 text/..
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0 (iPhone; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; GoogleBot/21.572; U; en) Presto/2.5.25 Version/10.54
2 image/..
1 text/..
3WikiBot/0.1
3 text/..
3Mozilla/5.0 (X11; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010091808 ThumbShotsBot (KFSW 3.0.6-3)
2 image/..
1 text/..
1 application/json
1 application/x-javascript
3AnomieBOT 1.0 (DeletionSortingCleaner)
3 application/json
9,653total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Generated on Tue, Jan 11, 2011 6:32
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.