Wikimedia Traffic Analysis Report - Crawler requests

Daily averages, based on sample period: 1 Aug 2011 - 31 Aug 2011
 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google
WMF traffic logging service suffered from server capacity problems in August and September 2011.
Absolute traffic counts for August 2011 are approximatly 6% too low.
Data loss only occurred during peak hours. It therefore may have had different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 64,037,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 396,462,000 external requests, which is 16.2%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
23,254google
18,976 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
1,633 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
607 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
461 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
218 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
152 code.google.com/p/crawler4j/text/..crawler4j (url)
127 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
99 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
99 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
83 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
75 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
70 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
65 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
65 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
57 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
54 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
41 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
40 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
37 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
24 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
22 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
20 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
18 code.google.com/appengineimage/..AppEngine-Google; (url; appid: tinysrc)
17 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
16 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
11 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
9 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
8 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygpxy)
8 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
7 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
7 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
7 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
6 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikidashboard)
5 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: s~ooohembed)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxworx)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: retimeme)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wagagate)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg)
4 code.google.com/appenginetext/..Wiki.java 0.24 AppEngine-Google; (url; appid: wikipediatools)
3 www.google.com/bot.htmlimage/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
14,846yahoo
9,771 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
3,255 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
1,424 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
130 misc.yahoo.com.cn/help.htmltext/..Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
66 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
39 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! DE Slurp; url)
21 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
18 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url)
18 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
16 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
14 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
14 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
13 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
10 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
9 misc.yahoo.com.cn/help.html-Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
6 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
6 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
3 help.yahoo.com/help/us/ysearch/slurpapplication/x-javascriptMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
3 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
14,002facebook
8,735 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
5,028 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
169 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
58 developers.facebook.comimage/..facebookplatform/1.0 (url)
6 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
5 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
6,713google?
6,115 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
201 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
182 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
135 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
20 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
13 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
12 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
11 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
7 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b3
5 www.google.com/bot.htmltext/..Mozilla/5.0(compatible;GoogleBot/2.1;url)
5 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5
3 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
5,650bing
3,861 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
1,776 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
5 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
4 www.bing.com/bingbot.htmapplication/vnd.php.serializedMozilla/5.0 (compatible; bingbot/2.0; url)
1,909naver
1,868 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
21 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
11 help.naver.com/customer_webtxt_02.jsptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
6 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
1,743yandex
1,414 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
197 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
53 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
51 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
15 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
3 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
3 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
1,484baidu
1,362 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
46 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
40 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
14 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
12 www.baidu.com/search/spider.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; Baiduspider/2.0; url)
3 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
1,218msn
586 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
244 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
186 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
85 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
81 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
19 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
7 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
3 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._ (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
398youdao
382 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
6 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
6 toolbar.youdao.com/image/..Youdao Toolbar (url)
393traslated
393 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
34080legs
290 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
44 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
4 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url;) Gecko/2008032620
313sblog
179 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
46 fulltext.sblog.cz/text/..SeznamBot/3.0-test (url)
43 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
39 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
3 fulltext.sblog.cz/-SeznamBot/3.0 (url)
310soso
298 help.soso.com/webspider.htmtext/..Sosospider(url)
9 help.soso.com/webspider.htm-Sosospider(url)
277entireweb
270 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
3 www.entireweb.com/about/search_tech/speedy_spider/-Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
262php
142 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
39 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
37 pear.php.net/text/..PEAR HTTP_Request class ( url )
36 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
5 pear.php.net/image/..PEAR HTTP_Request class ( url )
3 pear.php.net/package/http_request2text/..HTTP_Request2/2.0.0RC1 (url) PHP/5.3.2-1ubuntu4.9
254exabot
250 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
4 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
232diveintopython22
232 diveintopython22.org/text/..OpenAnything/1.0 url
200www.
153 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
27 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
16 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
4 www.image/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
195jike
136 shoulu.jike.com/spider.htmltext/..JikeSpider Mozilla/5.0 (compatible; JikeSpider; url)
33 shoulu.jike.com/spider.htmlimage/..JikeSpider Mozilla/5.0 (compatible; JikeSpider; url)
12 shoulu.jike.com/spider.htmltext/..gosospider Mozilla/5.0 (compatible; JIKESpider; url)
8 www.jike.com/spider.htmltext/..gosospider Mozilla/5.0 (compatible; JIKESpider; url)
3 shoulu.jike.com/spider.htmlimage/..gosospider Mozilla/5.0 (compatible; JIKESpider; url)
188yacy
55 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.35-30-generic; java 1.6.0_20; Europe/en) url
54 yacy.net/bot.htmltext/..yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
19 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.35-gentoo-r4; java 1.6.0_20; Europe/el) url
15 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_17; Etc/en) url
10 yacy.net/bot.htmltext/..yacybot (sciencenet/any; amd64 Linux 2.6.35-30-generic; java 1.6.0_20; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_21; Europe/de) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-23-server; java 1.6.0_24; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.38-8-generic; java 1.6.0_26; Europe/de) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.33.7-server-2mnb; java 1.6.0_18; Europe/fr) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.38-10-server; java 1.6.0_22; America/en) url
183majestic12
183 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.0; url)
165qwiki
164 qwiki.comtext/..Qwiki Fetcher (url)
158sogou
143 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
5 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
5 www.sogou.com/docs/help/webmasters.htm#07application/vnd.php.serializedSogou web spider/4.0(url)
3 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url)
157semager
148 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
7 www.semager.de/blog/semager-bots/application/jsonMozilla/5.0 (compatible; Semager/1.4; url)
155wikipedia
94 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.14.0 url
33 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
7 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.13.0 url
5 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.1.0 url
5 en.wikipedia.orgtext/..url
3 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle2/2.1.14 url
152mediawiki
150 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
152wwwgogetpapers
125 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
27 wwwgogetpapers.com/text/..User-Agent: GoGetPapersBot (url)
149toolserver
97 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
39 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
5 toolserver.org/~dispenser/text/..WebWikipedia Python (url)
3 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02
3 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
130bsurprised
130 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
113archive
110 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
110wordpress
12 arthur2rcasc.wordpress.comtext/..WordPress/MU; url
8 stradivariusconcerti.wordpress.comtext/..WordPress/MU; url
4 mannaismayaadventure.wordpress.comtext/..WordPress/MU; url
4 curtisnarimatsu.wordpress.comtext/..WordPress/MU; url
4 kterrl.wordpress.comtext/..WordPress/MU; url
98sf
33 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
32 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
32 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
91scoutjet
91 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
86enwp
71 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
9 enwp.org/User:KingpinBottext/..KingpinBot (url)
6 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
83sentymetr
42 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
41 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
76echonest
63 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
13 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
74goo
50 help.goo.ne.jp/contact/text/..goo wikipedia (url)
12 help.goo.ne.jp/help/article/1142/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
8 help.goo.ne.jp/help/article/1142/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
71daum
64 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
6 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
71flipboard
28 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
18 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
16 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
9 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
65jetbrains
34 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
31 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
65FeedBurner
64 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
64avantbrowser
33 www.avantbrowser.comtext/..Advanced Browser (url)
31 www.avantbrowser.comtext/..Avant Browser (url)
64kosmix
62 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
63feedshow
33 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
30 www.feedshow.comtext/..FeedshowOnline (url)
62newsgator
32 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
30 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
60emining
58 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
58federatedmedia
56 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
45labsparadigma
14 labsparadigma.com/application/x-wikiOpenAnything/1.0 url
14 labsparadigma.com/text/..OpenAnything/1.0 url
11 labsparadigma.org/text/..OpenAnything/1.0 url
6 labsparadigma.org/application/x-wikiOpenAnything/1.0 url
42Anonymouse
33 Anonymouse.org/text/..url (Unix)
9 Anonymouse.org/image/..url (Unix)
42textdigger
41 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
42sistrix
41 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url)
41z-add
39 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
40microsoft
40 academic.research.microsoft.com/text/..librabot/2.0 (url)
38ahrefs
38 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/1.0; url)
38fairshare
30 fairshare.cctext/..Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
4 fairshare.cctext/..Mozilla crawl/5.0 (compatible; fairshare.cc url)
37sitebot
37 www.sitebot.org/robot/text/..Mozilla/5.0 (compatible; SiteBot/0.1; url)
36apache
35 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address )
35tinyurl
33 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
34graemef
34 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
34SearchNearMe
26 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
8 SearchNearMe.com/contact.phptext/..SearchNearMe (url)
34it-influentials
34 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
34hatena
31 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
33ponderer
33 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
33zootycoon
33 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
32timewe
32 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
32ranchero
32 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
32rssbandit
32 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
32kula
32 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
32rssreader
32 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
32orcabrowser
32 www.orcabrowser.comtext/..Orca Browser (url)
31sgvlib
18 sgvlib.orgimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
13 sgvlib.orgtext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
31zipcommander
31 www.zipcommander.com/text/..1st ZipCommander (Net) - url
31snarfware
31 www.snarfware.com/text/..Snarfer/0.x.x (url)
31plagger
31 plagger.org/text/..Plagger/0.x.xx (url)
31blogbridge
31 www.blogbridge.com/text/..BlogBridge 2.13 (url)
31winpodder
31 winpodder.comtext/..WinPodder (url)
31nemui
31 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
31seebot
31 seebot.orgtext/..Lynx/2.8 (;url)
30feeds4all
30 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
30wikimedia
26 tools.wikimedia.de/~daniel/text/..WikiSense (url)
28dotnetdotcom
28 www.dotnetdotcom.org/text/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address )
26cdac
26 www.cdac.intext/..cdacp/Nutch-0.9 (IIT Kharagpur; url; mail address )
26suggy
26 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
26spinn3r
22 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
3 spinn3r.com/robot-Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
25alexa
24 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
25garlik
25 garlik.com/text/..GarlikCrawler/1.1 (url, mail address )
24puritysearch
24 www.puritysearch.net/text/..Mozilla/5.0 (compatible; Purebot/1.1; url)
24bibalex
15 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
9 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
24freebase
24 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
23whatrhymeswith
23 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
23turnitin
23 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url)
234chat
23 www.4chat.tvtext/..url
23yioop
19 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot url)
22github
10 github.com/NeilCrosby/wikislurpapplication/vnd.php.serializedWikiSlurp (url)
8 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
22tumblr
21 benderthewebrobot.tumblr.comtext/..Mozilla/5.0 (compatible; Bender; url)
22weblio
21 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
22moviecus
21 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url)
22goso
18 www.goso.cn/spider.htmltext/..gosospider Mozilla/5.0 (compatible; GOSOSpider; url)
4 www.goso.cn/spider.htmlimage/..gosospider Mozilla/5.0 (compatible; GOSOSpider; url)
22discoveryengine
18 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url)
4 discoveryengine.com/discobot.htmlimage/..Mozilla/5.0 (compatible; discobot/1.1; url)
21topsy
21 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
21kalooga
13 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url)
8 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url)
20globalspec
20 www.globalspec.com/Ocellitext/..Ocelli/1.4 (url)
20covario
20 www.covario.com/idstext/..Covario-IDS/1.0 (Covario; url; mail address )
19rcdtokyo
15 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
3 www.rcdtokyo.com/pc2m/image/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
19netnewswireapp
19 netnewswireapp.com/mac/-NetNewsWire/3.2.15 (Mac OS X; url; gzip-happy)
17froute
14 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
3 labs.froute.jp/pc2m/help.htmlimage/..Froute Mobile Gateway/1.0 (url)
17zapbot
6 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url)
6 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url)
5 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url)
16searchtechnologies
16 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
16rankur
16 rankur.comtext/..RankurBot/Rankur2.1 (url; mail address )
16rockpeaks
16 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
15drupal
10 drupal.org/text/..User-Agent: Drupal (url)
5 drupal.org/text/..Drupal (url)
15ibis
10 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
3 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
15search
15 www.search.ch/rim.htmltext/..UltraSpider3000/1.0 (url)
14bin-co
7 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
7 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
13123
13 www.123.fr/abus.htmltext/..PHP mutualise sur 123.fr - signalez les abus sur url
13yodao
13 www.yodao.com/help/webmaster/spider/text/..MozillaTest/5.0 (compatible; YodaoBot/1.0; url; )
13archive-it
9 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
4 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
13ac
13 ce.yazduni.ac.irtext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
13picsearch
12 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
12gnip
12 www.gnip.com/text/..UnwindFetchor/1.0 (url)
12dragonoff
9 www.dragonoff.com/php/dev/WikifySong.phpapplication/vnd.php.serializedurl
3 www.dragonoff.com/php/dev/WikifySong.phpapplication/jsonurl
12idrc
9 web.idrc.ca/challenge/ev-136691-201-1-DO_TOPIC.htmltext/..Mozilla/5.0 (compatible; http; url; mail address )
3 web.idrc.ca/challenge/ev-136691-201-1-DO_TOPIC.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; http; url; mail address )
11wise-guys
8 www.wise-guys.nl/text/..Mozilla/4.0 (compatible; Vagabondo/4.0/CGM; url)
11vik
10 vik.comtext/..vik-robot/Nutch-1.0 (vikspider; url; mail address )
10wikiglass
10 wikiglass.comtext/..url : mail address
10holmes
10 holmes.getext/..HolmesBot (url)
10paper
10 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
10js-kit
10 js-kit.com/text/..JS-Kit URL Resolver, url
10creativecommons
10 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
78,989total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
4,793PythonWikipediaBot/1.0
3,578 application/json
1,170 application/xml
45 text/..
1 -
1 image/..
1,281GoogleBot-Image/1.0
783 text/..
447 image/..
51 -
1,063MediaWikiCrawler-Google/2.0 ( mail address )
1,062 text/..
1 -
736php wikibot classes
701 application/vnd.php.serialized
35 text/..
1 -
1 application/json
649spider
648 text/..
1 image/..
1 -
1 application/ogg
448LinkParser/2.0
448 text/..
432Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
432 text/..
1 -
1 application/pdf
1 application/vnd.php.serialized
429Kavande Crawler 1.0/Nutch-1.4-dev (Iranian National Web Crawler)
429 text/..
1 -
1 image/..
335GoogleBot-Image/1.0
303 text/..
18 image/..
14 application/vnd.php.serialized
1 application/json
252wikiwix-bot-3.0
249 text/..
3 image/..
1 -
1 application/ogg
246Peachy MediaWiki Bot API Version 1.0
246 application/vnd.php.serialized
1 -
1 text/..
222Answersbot
222 text/..
176Onespot Crawler
134 application/json
40 text/..
2 -
169ClueBot/2.0
169 application/vnd.php.serialized
162GoogleBot-News
162 text/..
1 -
1 application/xml
141SiocWikiBot/1.0
132 application/vnd.php.serialized
9 text/..
140ClueBot/1.1
139 application/vnd.php.serialized
1 text/..
125jikespider "Mozilla/5.0
124 text/..
1 -
1 application/xml
1 application/ogg
122Pywikipediabot/2.0
122 application/json
118gsa-crawler (Enterprise; S5-KUKT7ERTD8NJB; mail address )
117 text/..
1 -
111Opera/8.01 (J2ME/MIDP; MXit WebBot/1.4.0.0) Opera Mini/3.1
92 application/vnd.wap.xhtml+xml
10 image/..
8 text/..
1 -
100DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
85 text/..
13 application/xml
2 image/..
1 audio/midi
77Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
77 text/..
1 image/..
1 application/ogg
1 application/vnd.php.serialized
72Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
49 image/..
23 text/..
1 -
1 application/json
1 application/x-javascript
68DNSTallyKwBot/0.2
68 text/..
65Mozilla/4.0 (compatible; EmberSpider 0.8; Scout (a); bgft)
65 text/..
63Test Webbot
63 text/..
59GoogleBot
59 text/..
1 image/..
55 mail address
54 application/vnd.php.serialized
1 text/..
51DotNetWikiBot/2.97 (Unix 5.10.0.0; )
26 application/xml
25 text/..
50YBot/0.1
50 application/vnd.php.serialized
48Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
48 text/..
1 -
45easycotes bot
45 text/..
43TVersity Media Robot
43 text/..
39DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
38 text/..
1 application/xml
37ROCKMELT-BOT
37 application/xml
1 text/..
35jikespider ("Mozilla/5.0)
35 text/..
1 -
1 application/ogg
35MediaWiki::Bot/3.2.6
35 application/json
1 -
33Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
33 text/..
1 -
33wikbot/1.21 CFNetwork/485.13.9 Darwin/11.0.0
19 image/..
14 application/json
1 -
1 text/..
31Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
31 text/..
31UCMore Crawler App
31 text/..
1 -
30Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
30 text/..
30AnomieBOT 1.0 (TagDater)
30 application/json
30SineBot/1.5.17(User:SineBot)
29 application/vnd.php.serialized
1 text/..
1 -
28MLBot (www.metadatalabs.com/mlbot)
17 text/..
11 application/vnd.php.serialized
28python-wikitools/1.2 (User:BernsteinBot)
28 application/json
27Mozilla/5.0 (compatible; en) Crawler from G51.
27 text/..
23Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
18 image/..
5 text/..
1 application/x-javascript
23WebCrawler/Nutch-1.2 (WebCrawler; WebCrawler)
23 text/..
1 image/..
1 application/ogg
22COIBot/2.0
22 text/..
20AnomieBOT 1.0 (ReplaceExternalLinks2)
20 application/json
1 text/..
20HRoestBot, de-wikipedia using pywikipedia framework
8 application/json
7 application/xml
5 text/..
19Peachy MediaWiki Bot API Version 0.1beta
19 application/vnd.php.serialized
19ibo2bot
19 text/..
19buzzbox bot
19 text/..
18FAST Enterprise Crawler 6 used by reedbusiness ( mail address )
18 text/..
18Twitterbot/0.1
17 text/..
1 image/..
1 -
18FAST Enterprise Crawler 6 used by ESP ( mail address )
18 text/..
17HTMLParser/1.6
17 text/..
1 -
17COIBot/1.00
17 text/..
17.NET Client Parser
17 application/xml
1 text/..
16GoogleBot/2.1
16 text/..
1 image/..
16Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
12 image/..
4 text/..
15Friendly Spider 1.0 contact mail address
15 text/..
15AnomieBOT 1.0 (TemplateSubster)
15 application/json
14AnomieBOT 1.0 (OrphanReferenceFixer)
14 application/json
1 text/..
14FAST Enterprise Crawler 6 used by test ( mail address )
14 text/..
1 -
13Twitterbot/1.0
13 text/..
1 image/..
1 application/ogg
13ReadonlyBot
13 text/..
13Happy OpenBuildings Robot
11 application/json
2 text/..
13FAST Enterprise Crawler 6 used by viaapia (viaapia)
12 text/..
1 -
12Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
12 text/..
1 -
1 application/rsd+xml
1 application/xml
12TrueKnowledgeBot bot mail address >
9 application/vnd.php.serialized
3 application/xml
11DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
11 text/..
1 application/xml
11TheKeens bot
11 text/..
11Mihas-Bot/0.1
11 application/vnd.php.serialized
10MyCuteBot/0.1
9 text/..
1 application/json
10SurakWare MediaWiki Bot/1.0
10 text/..
1 application/xml
10~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
10 text/..
10AnomieBOT 1.0 (BAGBot)
7 application/json
3 text/..
9Tawbot (public svn release; plwiki)
9 text/..
9AniBot/0.9 php/curl
9 application/vnd.php.serialized
1 -
9XLinkBot/1.00
9 text/..
9SkimWordsBot/1.0
9 text/..
8TrailsBot/Nutch-1.2
8 text/..
8Freebase Deathbot
8 text/..
8gosospider "Mozilla/5.0
8 text/..
8SiocWikiBot
8 text/..
7HTMLParser/2.0
7 text/..
7DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
7 text/..
1 application/xml
7My Bot
7 text/..
7DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
6 text/..
1 application/xml
7DotNetWikiBot/2.96 (Unix 5.10.0.0; )
4 text/..
3 application/xml
7CheMoBot/1.00
7 text/..
6Mozilla/5.0 QunarBot/1.0
5 text/..
1 image/..
6MystBot/1.5 fr libwww-perl/6.02
6 text/..
6NCrawler *custom*
6 text/..
6infraEnterprise v8 Web Crawler
6 -
1 text/..
6SWAT Crawler. AGH University project. In case of problem contact: mail address Thanks.
6 text/..
6HTMLParser/1.4
6 text/..
6python-wikitools/1.2 (User:LaraBot)
6 application/json
5DotNetWikiBot/2.97 (Microsoft Windows NT 6.1.7600.0; )
5 text/..
1 application/xml
5python-wikitools/1.2 (User:Mr.Z-bot)
5 application/json
5 mail address (Mozilla compatible)
5 text/..
1 image/..
5bitlybot
5 text/..
1 image/..
1 audio/midi
5wikbotlite/1.0 CFNetwork/485.13.9 Darwin/11.0.0
4 image/..
1 application/json
1 -
1 text/..
5DotNetWikiBot/2.9 (Unix 5.10.0.0; )
5 text/..
5TravelRecordBot/1.0
5 text/..
4Mozilla/5.0 (compatible; PaperLiBot/2.1)
4 text/..
4NutchRobot/Nutch-1.3
4 text/..
1 -
4wikbot/1.21 CFNetwork/485.12.7 Darwin/10.4.0
3 image/..
1 application/json
1 -
1 text/..
4AnomieBOT 1.0 (DeletionSortingCleaner)
4 application/json
4TwynCatBot/0.1 (Contact: www.twyn.com)
4 application/json
4Handelabra WikiBot
3 application/vnd.php.serialized
1 text/..
4Webwiki Search Engine Bot - www.webwiki.de
4 text/..
4GNAA-bot
4 text/..
4AnomieBOT 1.0 (RandomPagePicker)
4 application/json
4Geni ircpybot 1.0
2 application/json
2 text/..
1 application/xml
4Mozilla 5.0 (Apibot 0.31dev)
4 application/vnd.php.serialized
4DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
4 text/..
1 application/xml
3Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
3 application/xml
3Mozilla 5.0 (Apibot 0.31)
3 application/vnd.php.serialized
3FAST Enterprise Crawler 6 used by UBS ( mail address )
3 text/..
1 -
3TwengaBot-Discover
3 image/..
1 -
1 text/..
3Mozilla 5.0 (Apibot 0.30b5)
3 application/vnd.php.serialized
1 text/..
3Wikibot 1.21 (iPad; iPhone OS 4.2.1; en_US)
3 text/..
3PadosAttilaCrawler/Nutch-1.0 (Ozi,PolandWiz,AustriaWiz,WiennaWiz crawlers, Attila Pados, mail address ; www.ozi.hu, www.polandwiz.com,www.wiennawiz.com,www.austriawiz.com; attila dot mail address )
3 text/..
3wikbot/1.21 CFNetwork/485.12.30 Darwin/10.4.0
2 application/json
1 image/..
3MediaWiki::Bot/3.1.6 (User:Plasticspork)
3 application/json
3AnomieBOT 1.0 (AFDMergeFromCleaner)
3 application/json
3OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
3 text/..
1 -
3HBC Archive Indexerbot 0.9a
3 text/..
3unblockbot/1.00
3 text/..
3BotMapDev/1.3.587 CFNetwork/485.13.9 Darwin/11.0.0
3 image/..
1 text/..
3123peoplebot/1.0
3 text/..
3Dealer.com Robot 1.0
3 text/..
3Slevnicka.cz CURL bot
3 text/..
3Web Corpus Crawler
3 text/..
3QuickFinder Crawler
3 text/..
3SuperBot/4.7.0.74 (Windows XP)
3 text/..
1 image/..
3DotNetWikiBot/2.9 (Microsoft Windows NT 6.0.6000.0; )
3 text/..
3NFCCheckBot/1.0
3 text/..
3Wikibot 1.2 (Macintosh; Mac OS X 10.7.0; en_US_POSIX)
3 text/..
3Mozilla/5.0 (Bgbot 0.5)
3 text/..
3MR Crawler/Nutch-1.3
3 text/..
14,214total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Generated on Thu, Oct 6, 2011 14:54
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.