Wikimedia Traffic Analysis Report - Crawler requests

Daily averages, based on sample period: 1 Feb 2011 - 28 Feb 2011
 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 60,426,000 page requests (mime type text/html only!) per day are considered crawler requests, out of 412,719,000 external requests, which is 14.6%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
22,210google
17,134 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
1,218 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
838 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
701 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
464 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
306 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
222 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
97 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
75 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
72 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
60 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
56 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
49 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps)
39 code.google.com/p/crawler4j/text/..crawler4j (url)
35 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bobolaw1)
29 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
28 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mhomeroxy)
28 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975)
28 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995)
26 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 247-0062)
25 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy)
25 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat)
22 code.google.com/appenginetext/..AppEngine-Google; (url; appid: garawebsite)
22 code.google.com/appenginetext/..AppEngine-Google; (url; appid: suzetteklierocks)
21 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kmhsunblocker)
21 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
20 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aadyakshar)
19 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyflying0)
18 code.google.com/appenginetext/..AppEngine-Google; (url; appid: puthiyathiravidan)
17 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
17 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
17 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
15 code.google.com/appenginetext/..AppEngine-Google; (url; appid: maltingsproxy)
15 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gj-girgit)
14 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
13 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100thpriest)
12 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyflying1)
12 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: proxyflying0)
12 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
10 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ml-girgit)
10 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hydraroxy)
10 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findadvise)
10 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: oohembed)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur)
9 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: proxyflying2)
9 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url),gzip(gfe) AppEngine-Google; (http://code.google.com/appengine; appid: proxyflying1)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wsproxyserver)
9 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; url)
9 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jompr0xy)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler01)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nwikiproxy)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cmd-proxy)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: te-girgit)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server)
8 code.google.com/appenginetext/..AppEngine-Google; (url; appid: krittproxy)
7 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thakurproxy)
7 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
7 code.google.com/appenginetext/..AppEngine-Google; (url; appid: uber-proxy)
7 code.google.com/appenginetext/..AppEngine-Google; (url; appid: girgitiya)
6 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pa-girgit)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
6 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler00)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mrpiston123)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler04)
5 code.google.com/appenginetext/..AppEngine-Google; (url; appid: spq-popularity)
5 code.google.com/appengineimage/..AppEngine-Google; (url; appid: 100thpriest)
5 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
5 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
4 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
4 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: nwikiproxy)
4 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler02)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiki-crawler03)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: srcbackdoor)
4 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sebwebproxy)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mistakeproxyarea)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyflying2)
3 code.google.com/appenginetext/..oohEmbed.com AppEngine-Google; (url; appid: vipoembed)
3 sites.google.com/site/bendercrawlertext/..Mozilla/5.0 (compatible; Bender; url)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nethinguwnt)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vn-zoom)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: myanmarfamilyproxy)
3 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
3 code.google.com/appengineapplication/jsonPython-urllib/2.5 AppEngine-Google; (url; appid: loeschmonitor)
3 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hi-girgitiya)
18,854facebook
15,539 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
2,942 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
315 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
29 developers.facebook.comimage/..facebookplatform/1.0 (url)
15 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
10 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
3 developers.facebook.comtext/..facebookplatform/1.0 (url)
13,528yahoo
10,305 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
2,816 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
146 misc.yahoo.com.cn/help.htmltext/..Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
42 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! DE Slurp; url)
29 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
29 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
22 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
19 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
19 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url)
18 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
17 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlapplication/vnd.php.serializedY!J SearchMonkey/1.0 (Y!J-AGENT; url)
17 help.yahoo.com/help/us/ysearch/crawling/crawling-01.htmltext/..Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; url)
15 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
6 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
6 misc.yahoo.com.cn/help.html-Mozilla/5.0 (compatible; Yahoo! Slurp China; url)
5 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
5 www.yahoo.comtext/..fred (url)
4 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
5,488google?
5,063 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
188 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
85 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
29 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
26 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
25 www.google.com/bot.htmltext/..GoogleBot/2.1/Nutch-1.1 (url; http://www.google.com/bot.html; mail address )
22 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
22 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
17 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
3 www.google.com/bot.htmlapplication/jsonGoogleBot/2.1 (url)
3,957bing
2,872 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
1,052 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
29 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
2,091naver
1,994 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
65 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
14 help.naver.com/delete_main.asptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
8 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
8 help.naver.com/customer_webtxt_02.jsptext/..Mozilla/4.0 (compatible; NaverBot/1.0; url)
2,008yandex
1,714 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
168 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
44 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
27 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
21 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
14 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
7 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; url)
3 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexZakladki/3.0; Dyatel; url)
1,526msn
1,135 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)._
114 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
92 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
63 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
55 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
51 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
5 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
3 search.msn.com/msnbot.htmapplication/xmlmsnbot/2.0b (url)._
1,190baidu
644 www.baidu.jp/spider/text/..Baiduspider(url)
438 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
46 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
21 www.baidu.jp/spider/text/..BaiduImagespider(url)
20 www.baidu.jp/spider/text/..DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0;url)
9 www.baidu.com/search/spider.htm-Baiduspider(url)
7 www.baidu.jp/spider/-Baiduspider(url)
440epfl
420 parsa.epfl.chtext/..parsa/Nutch-1.1 (parsa; url; mail address )
20 parsa.epfl.chimage/..parsa/Nutch-1.1 (parsa; url; mail address )
413majestic12
404 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url)
3 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.3.3; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html))
380youdao
349 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
14 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
12 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YodaoBot/1.0; url; )
4 toolbar.youdao.com/image/..Youdao Toolbar (url)
298entireweb
292 www.entireweb.com/about/search_tech/speedy_spider/text/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
3 www.entireweb.com/about/search_tech/speedy_spider/-Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)
287traslated
287 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
274covario
200 www.covario.com/idstext/..Covario-IDS/1.0 (Covario; url; mail address )
71 www.covario.comtext/..Covario-IDS/1.0 (Covario; url; mail address )
269sblog
146 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
65 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
27 fulltext.sblog.cz/robot/text/..SeznamBot/2.0 (url)
14 fulltext.sblog.cz/text/..SeznamBot/3.0-beta (url)
13 fulltext.sblog.cz/text/..SeznamBot/3.0-alpha (url)
266exabot
186 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
50 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url)
15 www.exabot.com/go/robotimage/..Mozilla/5.0 (compatible; Exabot-Images/3.0; url)
9 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
5 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot-Images/3.0; url)
254php
113 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
50 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
45 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
39 pear.php.net/text/..PEAR HTTP_Request class ( url )
3 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.1 (url) PHP/5.3.2
3 pear.php.net/image/..PEAR HTTP_Request class ( url )
243wordpress
22 driwancybermuseum.wordpress.comtext/..WordPress/MU; url
22 arthur2rcasc.wordpress.comtext/..WordPress/MU; url
14 hongkongwillie.wordpress.comtext/..WordPress/MU; url
11 nikolaykot.wordpress.comtext/..WordPress/MU; url
9 josefboberg.wordpress.comtext/..WordPress/MU; url
7 nikolayko.wordpress.comtext/..WordPress/MU; url
6 worldwright.wordpress.comtext/..WordPress/MU; url
6 nikolaygeorgievkotev.wordpress.comtext/..WordPress/MU; url
5 kterrl.wordpress.comtext/..WordPress/MU; url
5 psconline.wordpress.comtext/..WordPress/MU; url
5 vaquous2011.wordpress.comtext/..WordPress/MU; url
4 tgbp.wordpress.comtext/..WordPress/MU; url
4 retroplayerbrazil.wordpress.comtext/..WordPress/MU; url
3 intheknow7.wordpress.comtext/..WordPress/MU; url
3 diesxdiemxdocet.wordpress.comtext/..WordPress/MU; url
3 mannaismayaadventure.wordpress.comtext/..WordPress/MU; url
3 klausgauger.wordpress.comtext/..WordPress/MU; url
3 antipsychoticdrugs.wordpress.comtext/..WordPress/MU; url
3 lovecraft1890.wordpress.comtext/..WordPress/MU; url
230toolserver
103 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
70 toolserver.org/~dispenser/text/..WebWikipedia Python/2.6 (url)
39 toolserver.org/~bayo/text/..LudoThecaire/1.0 (url)
14 toolserver.org/~guandalug/application/vnd.php.serializedGuandalugs PHPWikiBot/1.1 (url;de:User:Guandalug)
215yacy
23 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.18-194.17.4.el5xen; java 1.6.0_18; Etc/en) url
20 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.26-custom; java 1.6.0_22; Europe/en) url
14 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.9-023stab052.4-smp; java 1.6.0_23; GMT/de) url
14 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/nb) url
12 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-22-server; java 1.6.0_22; Europe/en) url
10 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_22; Europe/fr) url
9 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.9-023stab052.4-smp; java 1.6.0_22; GMT/de) url
9 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_24; Europe/en) url
8 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-29-generic; java 1.6.0_22; Europe/en) url
7 yacy.net/bot.htmltext/..yacybot (sciencenet/any; amd64 Linux 2.6.32-27-generic; java 1.6.0_20; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_22; Europe/en) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_24; Europe/fr) url
5 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.6.6; java 1.6.0_22; Europe/fr) url
5 yacy.net/bot.htmltext/..yacybot (sciencenet/any; amd64 Linux 2.6.32-28-generic; java 1.6.0_20; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_22; Europe/en) url
4 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.33.3-85.fc13.x86_64; java 1.6.0_18; Europe/fr) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_23-ea; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.18-028stab070.2; java 1.6.0_0; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_22; Europe/fr) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-28-generic; java 1.6.0_20; Europe/en) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_15; Europe/fr) url
3 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_23; Europe/de) url
208wikipedia
47 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.3.0 url
47 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WikiCleaner (url)
30 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.8.0 url
18 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.1.0 url
15 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.5.0 url
15 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.7.0 url
13 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.6.0 url
4 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.6 url
3 fr.wikipedia.org/wiki/Utilisateur:geobotapplication/jsongeobot, see url (uses Perl MediaWiki::API)
3 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/0.9.11 url
3 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
3 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.2.0 url
185scoutjet
185 www.scoutjet.com/text/..Mozilla/5.0 (compatible; ScoutJet; url)
170wikimedia
165 tools.wikimedia.de/~daniel/text/..WikiSense (url)
4 tools.wikimedia.de/~para/GeoCommons/text/..url
164enotes
84 www.enotes.comimage/..eNotesBot 2.0 (url)
80 www.enotes.comtext/..eNotesBot 2.0 (url)
150sogou
145 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
3 www.sogou.com/docs/help/webmasters.htm#07application/vnd.php.serializedSogou web spider/4.0(url)
146ayna
146 www.ayna.comtext/..Mozilla/5.0 (compatible; Ayna url)
142suggy
141 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
141sentymetr
72 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
69 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
125goo
107 help.goo.ne.jp/contact/text/..goo wikipedia (url)
7 help.goo.ne.jp/help/article/1142/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
5 help.goo.ne.jp/help/article/1142/application/xmlDoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
123soso
110 help.soso.com/webspider.htmtext/..Sosospider(url)
8 help.soso.com/soso-image-spider.htmtext/..Sosoimagespider(url)
3 help.soso.com/webspider.htm-Sosospider(url)
116www.
52 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
37 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
16 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
9 www.text/..Google - GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
115phonifier
115 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url)
96daum
96 ws.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/2.0
90freebase
87 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
3 www.freebase.com-metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
87sf
29 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
28 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
26 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
86z-add
79 w3.z-add.co.uk/linkcheck/text/..Z-Add Link Checker (url)
6 w3.z-add.co.uk/linkcheck/image/..Z-Add Link Checker (url)
83semager
70 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4; url)
12 www.semager.de/blog/semager-bots/application/jsonMozilla/5.0 (compatible; Semager/1.4; url)
82kosmix
74 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
8 www.kosmix.com/html/kosmos.htmltext/..Mozilla/5.0(compatible;Kosmos/1.0;url)
81archive-it
56 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
24 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible;archive.org_bot; Archive-It; url) Firefox/0.0
81FeedBurner
81 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
6980legs
45 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
22 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
69enwp
49 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
10 enwp.org/User:KingpinBottext/..KingpinBot (url)
9 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
59sitebot
58 www.sitebot.org/robot/text/..Mozilla/5.0 (compatible; SiteBot/0.1; url)
59jetbrains
30 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
28 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
57newsgator
26 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
26 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
3 www.newsgator.com/Individuals/NetNewsWire/-NetNewsWire/3.2.8 (Mac OS X; url; gzip-happy)
55emining
52 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
3 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address )
55avantbrowser
27 www.avantbrowser.comtext/..Advanced Browser (url)
27 www.avantbrowser.comtext/..Avant Browser (url)
54feedshow
27 www.feedshow.comtext/..FeedshowOnline (url)
27 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
46simplepie
25 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
18 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
44grouponia
29 www.grouponia.comtext/..WordPress/3.0.4; url
14 www.grouponia.comimage/..WordPress/3.0.4; url
42dailycouponds
29 dailycouponds.comtext/..WordPress/3.0.4; url
13 dailycouponds.comimage/..WordPress/3.0.4; url
41dotnetdotcom
41 www.dotnetdotcom.org/text/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address )
40rcdtokyo
34 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
5 www.rcdtokyo.com/pc2m/image/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
38hatena
35 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
3 mgw.hatena.ne.jp/helptext/..DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)
37Anonymouse
23 Anonymouse.org/text/..url (Unix)
14 Anonymouse.org/image/..url (Unix)
36dealgrater
25 dealgrater.comtext/..WordPress/3.0.4; url
11 dealgrater.comimage/..WordPress/3.0.4; url
33bibalex
22 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
11 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
33weblio
32 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
32zerd
24 www.zerd.net/tarantula/text/..Mozilla/5.0 (compatible; Tarantula; url)
8 www.zerd.net/text/..Mozilla/5.0 (compatible; Tarantula; url)
30it-influentials
30 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
29rssbandit
29 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
29ponderer
29 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
29graemef
29 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
29seebot
29 seebot.orgtext/..Lynx/2.8 (;url)
28tinyurl
28 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
28nguber
27 www.nguber.comtext/..g10_132_yy_xxx/101213 (Mesin Pencari bahasa Indonesia; url; mail address )
28zipcommander
28 www.zipcommander.com/text/..1st ZipCommander (Net) - url
28zootycoon
28 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
28timewe
28 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
28orcabrowser
28 www.orcabrowser.comtext/..Orca Browser (url)
28plagger
28 plagger.org/text/..Plagger/0.x.xx (url)
28nemui
28 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
27rssreader
27 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
27puritysearch
27 www.puritysearch.net/text/..Mozilla/5.0 (compatible; Purebot/1.1; url)
27mobileproxy
27 mobileproxy.mobitext/..Mozilla/5.0 (compatible; MobileSurf; url)
27winpodder
27 winpodder.comtext/..WinPodder (url)
27github
25 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
27feeds4all
27 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
26blogbridge
26 www.blogbridge.com/text/..BlogBridge 2.13 (url)
26snarfware
26 www.snarfware.com/text/..Snarfer/0.x.x (url)
26ranchero
26 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
25kula
25 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
25wiktionary
25 en.wiktionary.org/wiki/User:Rukhabotapplication/jsonRukhabot/0.1 (url)
25thesmespace
25 www.thesmespace.com/iphoneiquitytext/..Mozilla/5.0 (compatible; Iphoneiquity; url)
24textdigger
24 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
23ibis
11 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
10 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url)
23discoveryengine
23 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/1.1; url
22spinn3r
19 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
21fairshare
14 fairshare.cctext/..Mozilla/5.0 url (X11; FreeBSD i386; en-US; rv:1.2a) Gecko/20021021
5 fairshare.cctext/..Mozilla crawl/5.0 (compatible; fairshare.cc url)
21gulliway
17 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
4 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url)
21kalooga
12 www.kalooga.com/info.html?page=crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url)
9 www.kalooga.com/info.html?page=crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url)
21metamoji
21 www.metamoji.com/jp/crawler.htmltext/..Mozilla/5.0 (compatible; MetamojiCrawler/1.0; url
20whatrhymeswith
20 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
20openindex
16 www.openindex.io/text/..openindex.io/Openindex.io_Nutch-1.2 (Open source search service; url; mail address )
4 www.openindex.io/crawler.htmltext/..Nutch/Openindex.io_Nutch-1.2 (Open source search service; url; mail address )
19froute
16 labs.froute.jp/pc2m/help.htmltext/..Froute Mobile Gateway/1.0 (url)
3 labs.froute.jp/pc2m/help.htmlimage/..Froute Mobile Gateway/1.0 (url)
18buzzwordometer
18 www.buzzwordometer.com/text/..Buzzwordometer scanner (url) - dont take me too seriously!
18creativepulses
18 creativepulses.nltext/..CreativePulses Crawler (url)
18gnip
18 www.gnip.com/text/..UnwindFetchor/1.0 (url)
18flipboard
10 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
5 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
3 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
17gigablast
17 www.gigablast.com/spider.htmltext/..Gigabot/3.0 (url)
17alexa
17 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
17holmes
17 holmes.getext/..HolmesBot (url)
17ac
14 www.cse.iitb.ac.in/~vishaal_h4text/..vishaal/Nutch-0.9 (IIT Bombay; url; mail address )
17netnewswireapp
11 netnewswireapp.com/mac/-NetNewsWire/3.2.11 (Mac OS X; url; gzip-happy)
3 netnewswireapp.com/mac/-NetNewsWire/3.2.13 (Mac OS X; url; gzip-happy)
16topsy
16 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
16cityreview
16 www.cityreview.org/crawler/text/..Cityreview Robot (url)
16bsurprised
15 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
15loc
6 loc.govtext/..Mozilla/5.0 (compatible; loc-crawler/0.11.0 url)
5 loc.govimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
3 loc.govtext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
15mixi
8 mixi.jp/text/..mixi-mobile-converter/1.0 (url)
7 mixi.jp/image/..mixi-mobile-converter/1.0 (url)
14teesoft
6 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 5.1; [lang code]; rv:[..]) Gecko/.. etc (url)
3 www.teesoft.info/image/..Mozilla/5.0 (Windows; Windows NT 6.0; [lang code]; rv:[..]) Gecko/.. etc (url)
14opensourceconnections
14 www.opensourceconnections.comtext/..Mozilla/5.0 (compatible; heritrix/2.0.2 url)
14rockpeaks
14 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
13kana
9 www.kana.comimage/..Mozilla/5.0 (compatible; heritrix/2.0.1url)
4 www.kana.comtext/..Mozilla/5.0 (compatible; heritrix/2.0.1url)
12memidex
12 www.memidex.com/_bottext/..Mozilla/5.0 (compatible; Memibot/1.0; url )
12creativecommons
12 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url
11linguee
11 www.linguee.com/bottext/..Linguee Bot (url; mail address )
11syndicat
11 www.syndicat.com/text/..Clever-BOT/2.0.2b (url)
11picsearch
10 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
10sygol
10 www.sygol.comtext/..SygolBot url
10printful
6 printful.com/bot.htmltext/..Mozilla/5.0 (compatible; PrintfulBot/1.0; url)
4 printful.com/bot.htmlimage/..Mozilla/5.0 (compatible; PrintfulBot/1.0; url)
10aport
10 www.aport.ru/helptext/..Mozilla/5.0 (compatible; AportWorm/3.2; url)
79,097total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
6,676PythonWikipediaBot/1.0
4,838 application/json
1,795 application/xml
43 text/..
1 -
1 image/..
869GoogleBot-Image/1.0
548 text/..
278 image/..
43 -
1 application/pdf
538LinkParser/2.0
538 text/..
464php wikibot classes
416 application/vnd.php.serialized
48 text/..
1 -
433Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
432 text/..
1 -
1 application/pdf
1 application/ogg
1 application/vnd.php.serialized
356MediaWikiCrawler-Google/1.0
356 text/..
1 -
355Onespot Crawler
275 application/json
77 text/..
3 -
346ClueBot/1.1
346 application/vnd.php.serialized
1 text/..
342wikiwix-bot-3.0
328 text/..
13 image/..
1 -
260Answersbot
260 text/..
252GoogleBot-Image/1.0
241 text/..
6 image/..
5 application/vnd.php.serialized
1 -
226spider
216 text/..
9 application/json
1 image/..
218ExactusBot-v0.1
218 text/..
197Peachy MediaWiki Bot API Version 1.0
197 application/vnd.php.serialized
188gsa-crawler (Enterprise; S5-MS8QQPJ5BGWAA; mail address )
188 text/..
142Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
78 image/..
63 text/..
1 application/x-javascript
1 application/json
135Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
134 text/..
1 application/ogg
1 image/..
1 application/xml
1 audio/midi
122GoogleBot-News
122 text/..
1 -
117ClueBot/2.0
117 application/vnd.php.serialized
1 text/..
110HTMLParser/1.6
110 text/..
94EternalBot/0.2 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot>
94 text/..
86DotNetWikiBot/2.92 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
86 text/..
1 application/xml
85CorenSearchBot/1.5 en libwww-perl/5.834
85 text/..
75Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.8.0) Opera Mini/3.1
60 application/vnd.wap.xhtml+xml
11 image/..
4 text/..
1 -
64Test Webbot
64 text/..
1 -
64SiocWikiBot/1.0
60 application/vnd.php.serialized
4 text/..
63DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7600.0; )
54 text/..
8 application/xml
1 image/..
53COMODOspider/Nutch-1.0
51 text/..
2 image/..
1 -
44MLBot (www.metadatalabs.com/mlbot)
29 text/..
15 application/vnd.php.serialized
1 -
44Mozilla/4.0 (compatible; EmberSpider 0.8; Scout (a); bgft)
44 text/..
43Pywikipediabot/2.0
43 application/json
43TVersity Media Robot
43 text/..
40FAST Enterprise Crawler 6 used by My Company ( mail address )
38 text/..
2 application/x-wiki
1 -
1 application/x-javascript
1 application/xml
1 application/opensearchdescription+xml
37AnomieBOT 1.0 (TagDater)
37 application/json
36OpenText Semantic Navigation Crawler 1.1/Nutch-1.1
35 text/..
1 -
36SineBot/1.5.17(User:SineBot)
35 application/vnd.php.serialized
1 text/..
28HTMLParser/2.0
28 text/..
28Twitterbot/0.1
27 text/..
1 image/..
1 -
1 video/ogg
27Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
27 text/..
27Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
27 text/..
1 -
27GoogleBot
27 text/..
1 image/..
26UCMore Crawler App
26 text/..
1 -
25ibo2bot
25 text/..
25FAST Enterprise Crawler 6 used by test ( mail address )
19 -
6 text/..
1 application/rsd+xml
23DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; )
22 text/..
1 application/xml
1 -
1 image/..
23AnomieBOT 1.0 (ReplaceExternalLinks2)
23 application/json
22VWBot - CorenSearchBot/1.5 en derivative
22 text/..
21.NET Client Parser
21 application/xml
1 text/..
20AniBot/0.9 php/curl
20 application/vnd.php.serialized
1 -
1 text/..
20Peachy MediaWiki Bot API Version 0.1beta
20 application/vnd.php.serialized
1 -
1 text/..
19infraEnterprise v8 Web Crawler
18 -
1 text/..
19AnomieBOT 1.0 (BAGBot)
15 application/json
4 text/..
19Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
14 image/..
5 text/..
1 application/x-javascript
18AnomieBOT 1.0 (OrphanReferenceFixer)
18 application/json
17MediaWiki::Bot/3.2.6
17 application/json
1 -
16DotNetWikiBot/2.96 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
15 text/..
1 application/xml
1 -
1 image/..
15HRoestBot, de-wikipedia using pywikipedia framework
7 application/json
6 application/xml
2 text/..
15SONIVIS MediaWiki API Bot 0.1.3
15 text/..
15AnomieBOT 1.0 (TemplateSubster)
15 application/json
14COIBot/1.00
14 text/..
1 -
13~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
13 text/..
13Opera/8.01 (J2ME/MIDP; MXit WebBot/1.1.7.0) Opera Mini/3.1
6 application/vnd.wap.xhtml+xml
6 image/..
1 text/..
1 -
12Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
12 text/..
1 image/..
1 application/ogg
11 mail address
10 application/vnd.php.serialized
1 text/..
11COIBot/2.0
11 text/..
11FAST Enterprise Crawler 6 used by ESP ( mail address )
11 text/..
10('python-wikitools/1.2 (User:BernsteinBot)',)
10 application/json
10Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot
10 text/..
1 image/..
9TrueKnowledgeBot bot mail address >
6 application/vnd.php.serialized
3 application/xml
9python-wikitools/1.2 (User:Mr.Z-bot)
9 application/json
8Handelabra WikiBot
5 text/..
3 application/vnd.php.serialized
8XLinkBot/1.00
8 text/..
8TheKeens bot
8 text/..
8MystBot/1.5 fr libwww-perl/5.835
8 text/..
8SurakWare MediaWiki Bot/1.0
8 text/..
8Tsinghua AI Lab Robot 2.0
8 text/..
1 -
8KWSS Crawler Ver. 0.1
8 text/..
7Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
7 text/..
1 -
7FAST Enterprise Crawler/5.3.4 ( mail address )
7 text/..
7Mozilla/5.0 (X11; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010120923 ThumbShotsBot (KFSW 3.0.6-3)
5 image/..
2 text/..
1 application/x-javascript
7NATE.ROBOT Mozilla/5.0 (Windows; Windows NT 5.1; en-US) AppleWebKit/533.4 KHTML Chrome/5.0.375.125 Safari/533.4
7 text/..
7 mail address (Mozilla compatible)
7 text/..
6Mozilla/5.0 (compatible; 3F/ALL-PLA.NET webcrawler)
6 text/..
6Mozilla/5.0 (compatible; PaperLiBot/2.1)
6 text/..
1 image/..
1 application/vnd.php.serialized
6SiocWikiBot
6 text/..
6GNAA-bot
6 text/..
6Bot/WP/EN/Alex_Bakharev/AlexNewArtBot
6 text/..
6DotNetWikiBot/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
6 text/..
1 application/xml
6Geni ircpybot 1.0
3 application/json
3 text/..
6ExophoraSpider
6 text/..
6Xaldon WebSpider 2.7.b8
6 text/..
1 image/..
6unblockbot/1.00
6 text/..
6('python-wikitools/1.2 (User:LaraBot)',)
6 application/json
5bitlybot
5 text/..
1 image/..
1 application/ogg
5DotNetWikiBot/2.91 (Microsoft Windows NT 6.0.6001 Service Pack 1; )
4 text/..
1 application/xml
5Mozilla/5.0 QunarBot/1.0
5 text/..
1 -
5Tawbot (public svn release; plwiki)
5 text/..
1 -
5Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/23.377; U; es) Presto/2.5.25 Version/10.54
4 image/..
1 text/..
5gsa-crawler (Enterprise; T2-LXRKXYNZENSAA; mail address )
5 text/..
5Freebase Deathbot
5 text/..
5AnomieBOT 1.0 (AFDMergeFromCleaner)
5 application/json
5DotNetWikiBot/2.9 (Unix 5.10.0.0; )
5 text/..
5MediaWikiCrawler-Google/2.0 ( mail address )
5 text/..
1 -
5MediaWiki::Bot/3.1.6 (User:SporkBot)
5 application/json
5Mozilla/5.0 (Bgbot 0.5)
5 text/..
5AnomieBOT 1.0 (DeletionSortingCleaner)
5 application/json
4lssbot
4 text/..
4Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/23.348; U; es) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
4Wikimedia Images Experimental Crawler ( mail address )
3 text/..
1 image/..
4MoovidaBot/0.1
4 text/..
4DotNetWikiBot/2.95 (Microsoft Windows NT 5.1.2600 Service Pack 2; )
4 text/..
1 application/xml
4DotNetWikiBot/2.96 (Unix 5.10.0.0; )
3 application/xml
1 text/..
4Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214 (Windows; Windows NT 5.1; compatible; GoogleBot/23.390; U; es) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/23.334; en) Presto/2.5.25 Version/10.54
2 image/..
1 text/..
3QuickFinder Crawler
3 text/..
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/23.348; en) Presto/2.5.25 Version/10.54
2 image/..
1 text/..
3AnomieBOT 1.0 (RandomPagePicker)
3 application/json
3webcrawler
3 text/..
3GunaSpider
3 text/..
3HBC Archive Indexerbot 0.9a
3 text/..
3Opera/9.80 (J2ME/MIDP; Opera Mini/5.0(compatible; GoogleBot/23.377; en) Presto/2.5.25 Version/10.54
3 image/..
1 text/..
3Philip-bot
3 application/json
14,150total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Generated on Sat, Mar 5, 2011 14:55
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.