Wikimedia Traffic Analysis Report - Crawler requests

Daily averages, based on sample period: 1 Mar 2011 - 31 Mar 2011
 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 54,208,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 397,463,000 external requests, which is 13.6%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
51,180facebook
46,441 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
3,965 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
417 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
320 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
23 developers.facebook.comimage/..facebookplatform/1.0 (url)
8 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
4 developers.facebook.comtext/..facebookplatform/1.0 (url)
16,484google
12,780 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
808 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
665 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
650 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
336 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
316 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
204 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
99 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
77 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
68 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
64 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
23 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
22 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vn-zoom)
21 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
19 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
19 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
19 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
19 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps)
17 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
17 code.google.com/p/crawler4j/text/..crawler4j (url)
16 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gj-girgit)
14 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aadyakshar)
14 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
11 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; url)
10 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ml-girgit)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: te-girgit)
7 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: oohembed)
7 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
6 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
5 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
5 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
5 sites.google.com/site/bendercrawlertext/..Mozilla/5.0 (compatible; Bender; url)
5 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
4 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
4 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pa-girgit)
4 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
3 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: vipoembed)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: retimeme)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findadvise)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
3 code.google.com/appengineimage/..AppEngine-Google; (url; appid: alex2610ps)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: girgitiya)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
13,589yahoo
9,981 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
3,010 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
220 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
131 misc.yahoo.com.cn/help.htmltext/..Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
42 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! DE Slurp; url)
38 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
26 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
19 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url)
18 help.yahoo.com/help/us/ysearch/crawling/crawling-01.htmltext/..Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; url)
17 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
16 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
16 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
12 developer.yahoo.com/searchmonkey/useragentimage/..Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; url)
9 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
9 misc.yahoo.com.cn/help.html-Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
8 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
6 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
3 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible; Yahoo! Slurp; url)
3 developer.yahoo.com/searchmonkey/useragenttext/..Mozilla/5.0 (compatible; Yahoo! SearchMonkey 1.0; url)
4,814bing
3,504 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
1,284 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
25 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
4,629google?
4,275 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
126 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
94 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
40 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
27 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
24 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
19 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
13 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
2,047naver
1,948 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
78 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
12 help.naver.com/customer_webtxt_02.jsptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
6 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
1,856yandex
1,549 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
173 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
48 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
30 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
23 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
13 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
7 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url)
1,687msn
1,071 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
222 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
128 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
96 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
77 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
70 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
6 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
4 search.msn.com/msnbot.htmtext/..adidxbot/1.1 (url)
3 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._ (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
3 search.msn.com/msnbot.htmapplication/vnd.php.serializedmsnbot/2.0b (url)._
1,296baidu
630 www.baidu.jp/spider/text/..Baiduspider(url)
561 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
44 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
20 www.baidu.jp/spider/text/..DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0;url)
16 www.baidu.jp/spider/text/..BaiduImagespider(url)
14 www.baidu.com/search/spider.htm-Baiduspider(url)
5 www.baidu.jp/spider/-Baiduspider(url)
3 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
372traslated
372 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
336youdao
294 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
14 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
13 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
8 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
6 toolbar.youdao.com/image/..Youdao Toolbar (url)
296entireweb
290 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
291exabot
220 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
60 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
9 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
240soso
199 help.soso.com/webspider.htmtext/..Sosospider(url)
25 help.soso.com/webspider.htm-Sosospider(url)
13 help.soso.com/soso-image-spider.htmtext/..Sosoimagespider(url)
238php
86 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
65 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
58 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
24 pear.php.net/text/..PEAR HTTP_Request class ( url )
3 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.1 (url) PHP/5.3.2
233sblog
128 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
46 fulltext.sblog.cz/text/..SeznamBot/3.0-beta (url)
27 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
26 fulltext.sblog.cz/robot/text/..SeznamBot/2.0 (url)
3 fulltext.sblog.cz/-SeznamBot/3.0-beta (url)
230majestic12
229 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
206wikipedia
132 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.8.0 url
40 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
7 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.9.0 url
6 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.1.0 url
5 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.6 url
4 en.wikipedia.orgtext/..url
3 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
199sentymetr
101 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
98 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
186wordpress
12 arthur2rcasc.wordpress.comtext/..WordPress/MU; url
9 josefboberg.wordpress.comtext/..WordPress/MU; url
9 christopherboe.wordpress.comtext/..WordPress/MU; url
7 diesxdiemxdocet.wordpress.comtext/..WordPress/MU; url
7 kterrl.wordpress.comtext/..WordPress/MU; url
5 driwancybermuseum.wordpress.comtext/..WordPress/MU; url
5 mannaismayaadventure.wordpress.comtext/..WordPress/MU; url
5 musiquefreak.wordpress.comtext/..WordPress/MU; url
5 iwansuwandy.wordpress.comtext/..WordPress/MU; url
4 teachtoefl.wordpress.comtext/..WordPress/MU; url
4 cricketdiane.wordpress.comtext/..WordPress/MU; url
4 machikawaco.wordpress.comtext/..WordPress/MU; url
4 nikolaygeorgievkotev.wordpress.comtext/..WordPress/MU; url
3 churchofthecosmos.wordpress.comtext/..WordPress/MU; url
3 storeddata.wordpress.comtext/..WordPress/MU; url
163toolserver
108 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
38 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
8 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
5 toolserver.org/~dispenser/text/..WebWikipedia Python/2.6 (url)
3 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/5.835
160sogou
143 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
10 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url)
5 www.sogou.com/docs/help/webmasters.htm#07application/vnd.php.serializedSogou web spider/4.0(url)
138wikimedia
133 tools.wikimedia.de/~daniel/text/..WikiSense (url)
127goo
107 help.goo.ne.jp/contact/text/..goo wikipedia (url)
14 help.goo.ne.jp/help/article/1142/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
125yacy
18 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-28-generic; java 1.6.0_20; Europe/en) url
11 yacy.net/bot.htmltext/..yacybot (sciencenet/any; amd64 Linux 2.6.32-29-generic; java 1.6.0_20; Europe/en) url
8 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) url
8 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_24; Europe/en) url
7 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_23-ea; Europe/en) url
7 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-22-server; java 1.6.0_24; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.26-custom; java 1.6.0_22; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (sciencenet/any; amd64 Linux 2.6.32-30-generic; java 1.6.0_20; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_24; Europe/fr) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_22; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.24-28-generic; java 1.6.0_24; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-28-generic; java 1.6.0_24; Europe/fr) url
121enwp
111 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
7 enwp.org/User:KingpinBottext/..KingpinBot (url)
103z-add
97 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
6 w3.z-add.co.uk/linkcheck/image/..Z-Add Link Checker (url)
94www.
36 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
25 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
21 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
5 www.text/..Google - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
4 www.application/xmlGoogleBot/2.1 (urlGoogleBot.com/bot.html)
88semager
77 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
10 www.semager.de/blog/semager-bots/application/jsonMozilla/5.0 (compatible; Semager/1.4; url)
85archive-it
58 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
26 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
83daum
82 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
80sf
27 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
26 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
25 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
79FeedBurner
77 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
78phonifier
78 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url)
75kosmix
68 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
7 www.kosmix.com/html/kosmos.htmltext/..Mozilla/5.0(compatible;Kosmos/1.0;url)
74bibalex
49 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
25 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
74sitebot
72 www.sitebot.org/robot/text/..Mozilla/5.0 (compatible; SiteBot/0.1; url)
69suggy
69 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
69freebase
65 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
4 www.freebase.com-metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
59yunrang
58 www.yunrang.com/yrspider.htmltext/..yrspider Mozilla/5.0 (compatible; YRSpider; url)
58ayna
58 www.ayna.comtext/..Mozilla/5.0 (compatible; Ayna url)
56newsgator
27 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
26 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
3 www.newsgator.com/Individuals/NetNewsWire/-NetNewsWire/3.2.8 (Mac OS X; url; gzip-happy)
54jetbrains
27 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
27 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
52echonest
33 the.echonest.com/reader/application/jsonnestReader/0.3 (discovery; url; reader at echonest.com)
13 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
3 the.echonest.com/reader.htmlapplication/jsonnestReader/0.2 (discovery; url; reader at echonest.com)
3 the.echonest.com/reader.htmltext/..nestReader/0.2 (discovery; url; reader at echonest.com)
51emining
49 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
51avantbrowser
26 www.avantbrowser.comtext/..Avant Browser (url)
25 www.avantbrowser.comtext/..Advanced Browser (url)
50feedshow
26 www.feedshow.comtext/..FeedshowOnline (url)
24 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
47rcdtokyo
40 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
6 www.rcdtokyo.com/pc2m/image/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
4580legs
26 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
17 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
44thesmespace
44 www.thesmespace.com/iphoneiquitytext/..Mozilla/5.0 (compatible; Iphoneiquity; url)
40sosvia
24 www.sosvia.comimage/..Mozilla/5.0 (compatible; heritrix/1.12.1 url)
16 www.sosvia.comtext/..Mozilla/5.0 (compatible; heritrix/1.12.1 url)
38dotnetdotcom
38 www.dotnetdotcom.org/text/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address )
38hatena
35 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
3 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
34textdigger
33 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
33covario
33 www.covario.com/idstext/..Covario-IDS/1.0 (Covario; url; mail address )
30simplepie
17 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
10 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
29printful
17 printful.com/bot.htmltext/..Mozilla/5.0 (compatible; PrintfulBot/1.0; url)
12 printful.com/bot.htmlimage/..Mozilla/5.0 (compatible; PrintfulBot/1.0; url)
29gnip
28 www.gnip.com/text/..UnwindFetchor/1.0 (url)
28flipboard
10 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
8 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
5 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
5 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
27zootycoon
27 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
27winpodder
27 winpodder.comtext/..WinPodder (url)
27plagger
27 plagger.org/text/..Plagger/0.x.xx (url)
27ponderer
27 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
27it-influentials
27 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
26uk-laptop-battery
26 www.uk-laptop-battery.co.uk/blogtext/..WordPress/3.1; url
26timewe
26 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
26ranchero
26 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
26kula
26 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
26graemef
26 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
26nemui
26 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
25tinyurl
25 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
25blogbridge
25 www.blogbridge.com/text/..BlogBridge 2.13 (url)
25rssreader
25 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
25zipcommander
25 www.zipcommander.com/text/..1st ZipCommander (Net) - url
25snarfware
25 www.snarfware.com/text/..Snarfer/0.x.x (url)
25orcabrowser
25 www.orcabrowser.comtext/..Orca Browser (url)
25rssbandit
25 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
25feeds4all
25 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
25seebot
25 seebot.orgtext/..Lynx/2.8 (;url)
23weblio
21 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
23princexml
19 www.princexml.comimage/..Prince/7.1 (url)
4 www.princexml.comtext/..Prince/7.1 (url)
22bsurprised
21 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
21Anonymouse
13 Anonymouse.org/image/..url (Unix)
8 Anonymouse.org/text/..url (Unix)
21ynotshare
21 www.ynotshare.comapplication/jsonurl Bot
21alexa
21 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
21lth
21 combine.it.lth.se/text/..Combine/3 url
20froute
16 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
4 labs.froute.jp/pc2m/help.htmlimage/..Froute Mobile Gateway/1.0 (url)
20puritysearch
20 www.puritysearch.net/text/..Mozilla/5.0 (compatible; Purebot/1.1; url)
20zearch
20 zearch.metext/..ZearchSpider/Nutch-1.2 (Zearch Crawler (Nutch); url)
20spinn3r
18 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
20gov
12 www.nlb.gov.sgtext/..Mozilla/5.0 (compatible; heritrix/1.8.0 url)
5 pandora.nla.gov.au/crawl.htmltext/..Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.5-x url)
19whatrhymeswith
19 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
19gulliway
15 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
4 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
19github
16 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
18cityreview
17 www.cityreview.org/crawler/text/..Cityreview Robot (url)
18ibis
12 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
5 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
18edu
11 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url)
3 dis.sci.ntu.edu.sgtext/..momo/nutch-1.0 (momo; url; ye0001.ntu.edu.sg)
3 dis.sci.ntu.edu.sgtext/..momo/nutch-1.0 (momo; url; mail address )
18holmes
18 holmes.getext/..HolmesBot (url)
18netnewswireapp
13 netnewswireapp.com/mac/-NetNewsWire/3.2.15 (Mac OS X; url; gzip-happy)
5 netnewswireapp.com/mac/-NetNewsWire/3.2.14 (Mac OS X; url; gzip-happy)
16topsy
16 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
16enotes
16 www.enotes.comtext/..eNotesBot 2.0 (url)
16scoutjet
16 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
16apache
15 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address )
15mobileproxy
15 mobileproxy.mobitext/..Mozilla/5.0 (compatible; MobileSurf; url)
15discoveryengine
14 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url
15picsearch
13 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
14umamao
14 umamao.com/text/..UmamãoBot/0.1 (url)
14rockpeaks
14 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
13syndicat
13 www.syndicat.com/text/..Clever-BOT/2.0.2b (url)
13cydral
8 www.cydral.comtext/..CydralSpider/3.2 (Cydral Image Search; url)
5 www.cydral.comimage/..CydralSpider/3.2 (Cydral Image Search; url)
12wise-guys
10 www.wise-guys.nl/text/..Mozilla/4.0 (compatible; Vagabondo/4.0/CGM; url)
12creativecommons
12 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
11fairshare
5 fairshare.cctext/..Mozilla crawl/5.0 (compatible; fairshare.cc url)
3 fairshare.cctext/..Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
11linkedin
8 www.linkedin.comimage/..LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 url)
3 www.linkedin.comtext/..LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 url)
11drupal
5 drupal.org/text/..Drupal (url)
4 drupal.org/text/..User-Agent: Drupal (url)
11mixi
6 mixi.jp/text/..mixi-mobile-converter/1.0 (url)
5 mixi.jp/image/..mixi-mobile-converter/1.0 (url)
10teesoft
4 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
10search
10 www.search.ch/rim.htmltext/..UltraSpider3000/1.0 (url)
10kalooga
8 www.kalooga.com/info.html?page=crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url)
10creativepulses
10 creativepulses.nltext/..CreativePulses Crawler (url)
10vbseo
10 www.vbseo.comtext/..Mozilla/4.0 (vBSEO; url)
10swish-e
10 swish-e.org/text/..swish-e url
10wiktionary
10 en.wiktionary.org/wiki/User:Rukhabotapplication/jsonRukhabot/0.1 (url)
104,385total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
6,849PythonWikipediaBot/1.0
4,843 application/json
1,914 application/xml
92 text/..
1 -
812GoogleBot-Image/1.0
346 image/..
316 text/..
150 -
1 application/pdf
569php wikibot classes
497 application/vnd.php.serialized
72 text/..
1 -
522LinkParser/2.0
522 text/..
497ClueBot/1.1
497 application/vnd.php.serialized
401Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
400 text/..
1 -
1 application/pdf
1 application/vnd.php.serialized
339MediaWikiCrawler-Google/1.0
339 text/..
1 -
317Peachy MediaWiki Bot API Version 1.0
317 application/vnd.php.serialized
1 text/..
314wikiwix-bot-3.0
312 text/..
1 -
1 image/..
285Onespot Crawler
218 application/json
61 text/..
6 -
279GoogleBot-Image/1.0
261 text/..
10 application/vnd.php.serialized
8 image/..
1 -
267spider
267 text/..
1 image/..
251ExactusBot-v0.1
251 text/..
228Answersbot
228 text/..
152Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
150 text/..
1 image/..
1 application/ogg
1 -
1 application/xml
1 application/vnd.php.serialized
1 audio/midi
117GoogleBot-News
117 text/..
1 -
107ClueBot/2.0
107 application/vnd.php.serialized
90Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
62 image/..
28 text/..
1 application/json
1 application/x-javascript
85TVersity Media Robot
85 text/..
1 -
1 image/..
76MediaWikiCrawler-Google/2.0 ( mail address )
76 text/..
1 -
66CorenSearchBot/1.5 en libwww-perl/5.834
66 text/..
64gsa-crawler (Enterprise; S5-MS8QQPJ5BGWAA; mail address )
64 text/..
63HTMLParser/1.6
62 text/..
1 application/json
63Test Webbot
63 text/..
57COMODOspider/Nutch-1.0
55 text/..
2 image/..
1 -
1 application/ogg
53MLBot (www.metadatalabs.com/mlbot)
31 text/..
22 application/vnd.php.serialized
1 image/..
47Mozilla/4.0 (compatible; EmberSpider 0.8; Scout (a); bgft)
47 text/..
46SiocWikiBot
46 text/..
44ibo2bot
44 text/..
43FAST Enterprise Crawler 6 used by My Company ( mail address )
43 text/..
1 -
1 application/x-javascript
1 application/x-wiki
1 application/rsd+xml
1 application/opensearchdescription+xml
42 mail address
41 application/vnd.php.serialized
1 text/..
39AnomieBOT 1.0 (TagDater)
39 application/json
38Pywikipediabot/2.0
38 application/json
37DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7600.0; )
30 text/..
6 application/xml
1 image/..
35SineBot/1.5.17(User:SineBot)
34 application/vnd.php.serialized
1 text/..
31YBot/0.1
31 application/vnd.php.serialized
30Opera/8.01 (J2ME/MIDP; MXit WebBot/1.2.0.0) Opera Mini/3.1
25 application/vnd.wap.xhtml+xml
3 image/..
2 text/..
1 -
29DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
29 text/..
1 application/xml
28DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
23 text/..
5 application/xml
1 image/..
28Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.8.0) Opera Mini/3.1
22 application/vnd.wap.xhtml+xml
4 image/..
2 text/..
1 -
27Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
27 text/..
27MediaWiki::Bot/3.2.6
27 application/json
26Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
26 text/..
1 -
26python-wikitools/1.2 (User:Mr.Z-bot)
26 application/json
26OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
23 text/..
3 -
26UCMore Crawler App
26 text/..
26DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
26 text/..
1 application/xml
25GoogleBot
25 text/..
1 image/..
23EternalBot/0.2 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot>
23 text/..
22AnomieBOT 1.0 (ReplaceExternalLinks2)
22 application/json
22Tawbot (public svn release; plwiki)
22 text/..
20Twitterbot/0.1
20 text/..
1 -
1 image/..
19AnomieBOT 1.0 (BAGBot)
16 application/json
3 text/..
19VWBot - CorenSearchBot/1.5 en derivative
19 text/..
18DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
18 text/..
1 application/xml
18Peachy MediaWiki Bot API Version 0.1beta
18 application/vnd.php.serialized
18Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
14 image/..
4 text/..
1 application/json
1 application/x-javascript
18.NET Client Parser
18 application/xml
1 text/..
16AnomieBOT 1.0 (OrphanReferenceFixer)
16 application/json
16FAST Enterprise Crawler 6 used by ESP ( mail address )
16 text/..
15COIBot/1.00
15 text/..
14Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.743; U; es) Presto/2.5.25 Version/10.54
11 image/..
3 text/..
1 application/x-javascript
14HRoestBot, de-wikipedia using pywikipedia framework
6 application/json
5 application/xml
3 text/..
14Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
14 text/..
1 -
1 image/..
14AnomieBOT 1.0 (TemplateSubster)
14 application/json
11COIBot/2.0
11 text/..
10Mozilla/5.0 (compatible; PaperLiBot/2.1)
10 text/..
1 application/pdf
1 image/..
10HTMLParser/2.0
9 text/..
1 -
10SurakWare MediaWiki Bot/1.0
10 text/..
1 application/xml
9trunk.ly spider mail address
9 text/..
1 image/..
9infraEnterprise v8 Web Crawler
9 -
9AniBot/0.9 php/curl
9 application/vnd.php.serialized
1 image/..
9MyCuteBot / 0.1.
9 text/..
9MoovidaBot/0.1
9 text/..
9NATE.ROBOT Mozilla/5.0 (Windows; Windows NT 5.1; en-US) AppleWebKit/533.4 KHTML Chrome/5.0.375.125 Safari/533.4
9 text/..
9Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/23.411; U; es) Presto/2.5.25 Version/10.54
7 image/..
2 text/..
9AfDstats query by User:Staeiou, trying to restart AfDStatBot
9 application/vnd.php.serialized
8('python-wikitools/1.2 (User:BernsteinBot)',)
8 application/json
8TrueKnowledgeBot bot mail address >
4 application/xml
4 application/vnd.php.serialized
8DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 2; )
8 text/..
1 application/xml
8FAST Enterprise Crawler/5.3.4 ( mail address )
8 text/..
8Handelabra WikiBot
5 application/vnd.php.serialized
3 text/..
8XLinkBot/1.00
8 text/..
7DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7600.0; )
7 text/..
1 application/xml
1 image/..
7ATWspider/1.1
7 text/..
7Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/23.390; U; es) Presto/2.5.25 Version/10.54
5 image/..
2 text/..
7 mail address (Mozilla compatible)
7 text/..
1 image/..
7Jbot
7 text/..
6Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.741; U; es) Presto/2.5.25 Version/10.54
5 image/..
1 text/..
6TheKeens bot
6 text/..
6Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/24.732; U; es) Presto/2.5.25 Version/10.54
5 image/..
1 text/..
6Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot
6 text/..
1 image/..
6Geni ircpybot 1.0
3 application/json
3 text/..
1 application/xml
6('python-wikitools/1.2 (User:LaraBot)',)
6 application/json
5Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/23.405; U; es) Presto/2.5.25 Version/10.54
4 image/..
1 text/..
5Freebase Deathbot
5 text/..
5Catbot/0.0 ( mail address ;es;en)
5 text/..
5unblockbot/1.00
5 text/..
5bitlybot
5 text/..
1 -
1 image/..
5MystBot/1.5 fr libwww-perl/5.835
5 text/..
5DotNetWikiBot/2.9 (Unix 5.10.0.0; )
5 text/..
4Jabse.com Crawler v.2.0 www.jabse.com/crawler.php
4 text/..
1 application/xml
4Bot/WP/EN/Alex_Bakharev/AlexNewArtBot
4 text/..
4Minisearch_Spider
4 text/..
1 -
4gsa-crawler (Enterprise; S5-NUKHP4PG4EJAT; mail address )
4 text/..
4AnomieBOT 1.0 (AFDMergeFromCleaner)
4 application/json
4Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/24.743; en) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
4Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/23.390; en) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
4DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
4 text/..
1 image/..
1 application/xml
4musiccrawl/1.0
4 text/..
4DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; )
4 text/..
1 -
1 application/xml
4DotNetWikiBot/2.96 (Unix 5.10.0.0; )
3 application/xml
1 text/..
4Synthesio Crawler release MonaLisa ( mail address )
4 text/..
4TextBot 0.2
4 text/..
1 -
4Mozilla/5.0 (Bgbot 0.5)
4 text/..
3ReadonlyBot
3 text/..
3Friendly Spider 1.0 contact mail address
3 text/..
3Mozilla/5.0 QunarBot/1.0
3 text/..
1 -
3UniFind Site Spider; email mail address
3 text/..
1 -
3yolinkBot
3 text/..
3HBC Archive Indexerbot 0.9a
3 text/..
3gsa-crawler (Enterprise; T2-R5QNHLX72WWBK; mail address )
3 text/..
3Teragram/SAS Crawler
3 text/..
1 application/rsd+xml
1 image/..
1 application/opensearchdescription+xml
1 application/xml
3DotNetWikiBot/2.94 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
2 text/..
1 application/xml
3AnomieBOT 1.0 (DeletionSortingCleaner)
3 application/json
3~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
3 text/..
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/23.411; en) Presto/2.5.25 Version/10.54
2 image/..
1 text/..
1 application/x-javascript
3ReapETbot/0.2 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot>
3 text/..
3AnomieBOT 1.0 (RandomPagePicker)
3 application/json
3gsa-crawler (Enterprise; T2-LXRKXYNZENSAA; mail address )
3 text/..
3FAST Enterprise Crawler 6 used by test ( mail address )
3 -
1 text/..
3Jyxobot/1
3 text/..
3IssueCrawler
3 text/..
14,328total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Generated on Wed, Apr 20, 2011 15:09
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.