Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 May 2012 - 31 May 2012 (last 12 months)

This analysis is based on a 1:1000 sampled server log (squids)

See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot
For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'
In total 67,897,030 page requests (mime type text/html only!) per day are considered crawler requests, out of 478,729,940 external requests, which is 14.2%

Page requests for crawlers that specify a url in the agent string

Count
x 1000 Secondary domain
(~site) name URL Mime type User agent

google

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html - Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html text/.. Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)

desktop.google.com/ application/xml Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)

www.google.com/bot.html - Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html image/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

desktop.google.com/ image/.. Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)

www.google.com/bot.html text/.. DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

desktop.google.com/ text/.. Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)

www.google.com/bot.html text/.. SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/feedfetcher.html - FeedFetcher-Google; (url)

www.google.com/feedfetcher.html image/.. Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

www.google.com/feedfetcher.html application/json Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

www.google.com/feedfetcher.html text/.. Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ortografia4)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wikien4)

www.google.com/feedfetcher.html application/xml FeedFetcher-Google; (url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ortopedianew)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~cloudcrawling)

code.google.com/p/crawler4j/ text/.. crawler4j (url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wikien3)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: rarplayer)

www.google.com/feedfetcher.html text/.. FeedFetcher-Google; (url)

www.google.com/feedfetcher.html application/xml Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

code.google.com/appengine image/.. AppEngine-Google; (url; appid: s~senchaiosrc)

code.google.com/appengine application/json AppEngine-Google; (url; appid: s~redconceptual)

desktop.google.com/ - Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)

docs.google.com image/.. Mozilla/5.0 (compatible; GoogleDocs; conversion; url)

code.google.com/appengine application/xml AppEngine-Google; (url; appid: wikipedia-raw)

www.google.com/feedfetcher.html - Mozilla/5.0 (compatible) FeedFetcher-Google; (url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: pakgalaxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: myproxywx)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: usawebdl)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: abdulfat)

www.google.com/coop/cse/cref text/.. FeedFetcher-Google-CoOp; (url)

code.google.com/appengine text/.. WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)

docs.google.com image/.. Mozilla/5.0 (compatible; GoogleDocs; documents; url)

code.google.com/appengine text/.. Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kires-roxy)

www.google.com/bot.html text/.. GoogleBot/2.1 (url)

www.google.com/feedfetcher.html text/.. Mozilla/5.0 (compatible) FeedFetcher-Google;(url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~wikigraph2)

code.google.com/appengine image/.. Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: bie99miracle)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxy-devakishor)

code.google.com/appengine text/.. www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)

code.google.com/appengine application/json MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~keytanwiki4)

www.google.com/feedfetcher.html image/.. Mozilla/5.0 (compatible) FeedFetcher-Google;(url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~keytanwiki2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~deutiki)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~drizzlprox)

docs.google.com image/.. Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)

code.google.com/appengine image/.. AppEngine-Google; (url; appid: d24-img)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: atxproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: nation4india)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: vi-mobile)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~keytanwiki3)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tusawebproxy4)

code.google.com/p/ronda application/json Ronda - url

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlp6)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: d24-img)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: worldwide-propaganda)

desktop.google.com/ application/xml Mozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlp9)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: free-data)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: nagarajhubli-proxy-server)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webponline0)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: 114proxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: chris-homework-helper)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webponline9)

docs.google.com text/.. Mozilla/5.0 (compatible; GoogleDocs; conversion; url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: simple-tools2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: vebproxy)

code.google.com/appengine - AppEngine-Google; (url; appid: s~senchaiosrc)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~zagrobelnyprox)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: proxyusing121)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: maltingsproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: pazvantoff)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~harunakaze)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: dkoxyserv)

desktop.google.com/ image/.. Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)

code.google.com/appengine text/.. Mozilla/5.0 AppEngine-Google; (url; appid: s~app3123ak)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlp8)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: paradigm-web-proxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~kyaysarlay)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: adrianswebproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~proxyseekkety)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: betafxserver)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: web-proxy-hh)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: pox)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: weps002)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wwwwebp0)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tmobile-internet)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: 9329269)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: hideproxyz)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: yuricamara)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: 9000tunnels)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: taterproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wwwwebp2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webproxy8-2)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: usawebdl3)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ivegotalovelybunch)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: simple-tools6)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~crowdsurfer100)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: thetechnolust)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: jptaravellahighschool)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: nhsportal)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: gaucho-labnol)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: python-proxy-server)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: nothing-unusual)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ideserveinternet)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: misc-tools)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wiwohk-proxy-server)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wagagate)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~francetiki)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ieultimateproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~keytanwiki1)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kbworld24)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: davidgotmoney50)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: openeyeproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: threewiki)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webproxy8-9)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: azamasmadi)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: predictionwrong)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ageryder)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: sony-words)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kaveriselvaraj)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: sizzsurf)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: mboharsik)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: dex-dwds)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: web4proxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: ivankrisproxyserver)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kuryproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: burmansearch)

www.google.com/feedfetcher.html image/.. FeedFetcher-Google; (url)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~misterhac)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: downarchivestuffproxyserver)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: cachehew)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: discretepword)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~link123451)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webproxy8-5)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~adddon1)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: web-mastered)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: quigonjinn03)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~japantiki)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: boxapp)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: kerouanen)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: laaplicaciondelucas)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~tyangbanga)

code.google.com/appengine text/.. Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webponline5)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: wwwwebp8)

code.google.com/appengine application/json AppEngine-Google; (url; appid: prfleme)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webusadlq0)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: webponline8)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: seiyukyouen)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~nyinayminproxy2)

www.google.com/bot.html NONE/wikipedia - Mozilla/5.0 (compatible; GoogleBot/2.1; url)

code.google.com/appengine text/.. oohEmbed.com AppEngine-Google; (url; appid: vipoembed)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: djshan45)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tgbeeson)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: mehproxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: azzaziprxy)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: tinkernutsearch)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: s~sony-hack)

code.google.com/appengine text/.. Wiki.java 0.25 AppEngine-Google; (url; appid: wikipediatools)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: trabserver)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: futureducation1)

code.google.com/appengine text/.. AppEngine-Google; (url; appid: web-phpproxy)

facebook

www.facebook.com/externalhit_uatext.php image/.. facebookexternalhit/1.0 (url)

www.facebook.com/externalhit_uatext.php - facebookexternalhit/1.0 (url)

www.facebook.com/externalhit_uatext.php text/.. facebookexternalhit/1.0 (url)

www.facebook.com/externalhit_uatext.php text/.. facebookexternalhit/1.1 (url)

developers.facebook.com image/.. facebookplatform/1.0 (url)

developers.facebook.com - facebookplatform/1.0 (url)

www.facebook.com/externalhit_uatext.php image/.. facebookexternalhit/1.1 (url)

developers.facebook.com text/.. facebookplatform/1.0 (url)

www.facebook.com/externalhit_uatext.php - facebookexternalhit/1.1 (url)

www.facebook.com/externalhit_uatext.php application/vnd.php.serialized facebookexternalhit/1.1 (url)

bing

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm - Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm application/vnd.php.serialized Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3

www.bing.com/bingbot.htm image/.. Mozilla/5.0 (compatible; bingbot/2.0; url)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html))

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: images-jpg)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) (via babelfish.yahoo.com)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible;bingbot/2.0;url)

www.bing.com/bingbot.htm text/.. Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: p8roxy)

google?

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html text/.. GoogleBot/2.1 (url)

www.google.com/bot.html application/vnd.php.serialized Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html image/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html - Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html - Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html text/.. Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html application/xml Mozilla/5.0 (compatible; GoogleBot/2.1; url)

www.google.com/bot.html text/.. Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html text/.. DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)

www.google.com/bot.html text/.. SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)

baidu

www.baidu.com/search/spider.html text/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.htm text/.. Baiduspider-image(url)

www.baidu.com/search/spider.html - Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.htm image/.. Baiduspider-image(url)

www.baidu.com/search/spider.html image/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.html application/vnd.php.serialized Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.html text/.. Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: misc-tools)

www.baidu.com/search/spider.html application/ogg Mozilla/5.0 (compatible; Baiduspider/2.0; url)

www.baidu.com/search/spider.htm - Baiduspider-image(url)

www.baidu.com/search/spider.htm text/.. Baiduspider(url)

yandex

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexBot/3.0; url)

yandex.com/bots - Mozilla/5.0 (compatible; YandexBot/3.0; url)

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexDirect/3.0; url)

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexImages/3.0; url)

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexImages/3.0; url)

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)

yandex.com/bots application/vnd.php.serialized Mozilla/5.0 (compatible; YandexBot/3.0; url)

yandex.com/bots image/.. Mozilla/5.0 (compatible; YandexBot/3.0; url)

yandex.com/bots text/.. Mozilla/5.0 (compatible; YandexNewslinks; url)

yandex.com/bots - Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)

naver

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url)

help.naver.com/robots/ - Yeti/1.0 (NHN Corp.; url)

help.naver.com/robots/ image/.. Yeti/1.0 (NHN Corp.; url)

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3

help.naver.com/robots/ text/.. Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5

corp.naver.jp/ text/.. Mozilla/5.0 (compatible; NaverJapan/1.0; url)

yahoo

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Slurp; url)

help.yahoo.com/help/us/ysearch/slurp image/.. Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. 'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)

help.yahoo.com/help/us/ysearch/slurp - Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)

listing.yahoo.co.jp/support/faq/int/other/other_001.html text/.. Y!J-BRJ/YATS crawler (url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html - 'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'

help.yahoo.com/help/us/ysearch/slurp - Mozilla/5.0 (compatible; Yahoo! Slurp; url)

developer.yahoo.com/yql/provider text/.. Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html image/.. 'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'

help.yahoo.com/help/us/ysearch/slurp application/vnd.php.serialized Mozilla/5.0 (compatible Yahoo! Slurp/3.0 url)

help.yahoo.com/help/us/ysearch/slurp text/.. Mozilla/5.0 (compatible; Yahoo! Nano; url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. Y!J-BRI/0.0.1 crawler ( url )

help.yahoo.com text/.. Mozilla/5.0 (YahooYSMcm/3.0.0; url)

help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html text/.. Y!J-BRT/1.0 crawler (url)

help.yahoo.com/help/us/ysearch/slurp application/json Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)

80legs

www.80legs.com/webcrawler.html text/.. Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620

www.80legs.com/webcrawler.html - Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620

www.80legs.com/webcrawler.html image/.. Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620

msn

search.msn.com/msnbot.htm text/.. msnbot/2.0b (url)

search.msn.com/msnbot.htm text/.. msnbot-media/1.1 (url)

search.msn.com/msnbot.htm text/.. msnbot-Products/1.0 (url)

search.msn.com/msnbot.htm image/.. msnbot-media/1.1 (url)

search.msn.com/msnbot.htm text/.. msnbot-UDiscovery/2.0b (url)

search.msn.com/msnbot.htm text/.. msnbot-NewsBlogs/2.0b (url)

search.msn.com/msnbot.htm - msnbot-media/1.1 (url)

search.msn.com/msnbot.htm - msnbot/2.0b (url)

search.msn.com/msnbot.htm application/vnd.php.serialized msnbot-media/1.1 (url)

search.msn.com/msnbot.htm - msnbot/0.01 (url)

blekko

blekko.com/about/blekkobot text/.. Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)

blekko.com/about/blekkobot - Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)

sblog

fulltext.sblog.cz/screenshot/ image/.. Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)

fulltext.sblog.cz/ text/.. SeznamBot/3.0 (url)

fulltext.sblog.cz/screenshot/ text/.. Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)

fulltext.sblog.cz/ - SeznamBot/3.0 (url)

cibra

cibra.de/ text/.. CiBra Data Collector (url)

ahrefs

ahrefs.com/robot/ text/.. Mozilla/5.0 (compatible; AhrefsBot/3.0; url)

ahrefs.com/robot/ - Mozilla/5.0 (compatible; AhrefsBot/3.0; url)

ahrefs.com/robot/ text/.. Mozilla/5.0 (compatible; AhrefsBot/2.0; url)

ahrefs.com/robot/ application/ogg Mozilla/5.0 (compatible; AhrefsBot/3.0; url)

www.

www. text/.. GoogleBot/2.1 ( urlGoogleBot.com/bot.html)

www. text/.. GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)

www. text/.. GoogleBot/2.1 (urlGoogleBot.com/bot.html)

php

pear.php.net/ application/vnd.php.serialized PEAR HTTP_Request class ( url )

pear.php.net/package/http_request2 text/.. HTTP_Request2/0.5.2 (url) PHP/5.2.17

pear.php.net/ text/.. PEAR HTTP_Request class ( url )

pear.php.net/package/http_request2 application/xml HTTP_Request2/2.0.0 (url) PHP/5.3.8

pear.php.net/ image/.. PEAR HTTP_Request class ( url )

pear.php.net/package/http_request2 text/.. HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.14

pear.php.net/ application/xml PEAR HTTP_Request class ( url )

sogou

www.sogou.com/docs/help/webmasters.htm#07 text/.. Sogou web spider/4.0(url)

www.sogou.com/docs/help/webmasters.htm#07 - Sogou web spider/4.0(url)

www.sogou.com/docs/help/webmasters.htm#07 image/.. Sogou Pic Spider/3.0(url)

www.sogou.com/docs/help/webmasters.htm#07 application/vnd.php.serialized Sogou web spider/4.0(url)

www.sogou.com/docs/help/webmasters.htm#07 text/.. Sogou Pic Spider/3.0(url)

majestic12

www.majestic12.co.uk/bot.php? text/.. Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)

www.majestic12.co.uk/bot.php? text/.. Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)

echonest

the.echonest.com/reader/ application/xml nestReader/0.3 (discovery; url; reader at echonest.com)

the.echonest.com/reader/ text/.. nestReader/0.3 (discovery; url; reader at echonest.com)

wordpress

klausgauger.wordpress.com text/.. WordPress/3.4-beta4-20825; url

klima47.wordpress.com text/.. WordPress/3.4-beta4-20725; url

kingcrimsonprog.wordpress.com image/.. WordPress/3.4-beta4-20766; url

wikipedia

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.19.0 url

en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation text/.. WikiCleaner (url)

en.wikipedia.org/wiki/User_talk:Blevintron text/.. BlevintronBot version 2012-05-19 contact url

en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation text/.. WPCleaner (url)

en.wikipedia.org/wiki/User_talk:Blevintron text/.. BlevintronBot version 2012-05-16 contact url

en.wikipedia.org/wiki/User_talk:Blevintron text/.. BlevintronBot version 2012-05-09 contact url

fr.wikipedia.org/wiki/Utilisateur:Salebot application/json Salebot, see url (uses Perl MediaWiki::API)

sk.wikipedia.org/wiki/Redaktor:TeslaBot - TeslaBot (url)

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.18.0 url

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.19.2 url

en.wikipedia.org/wiki/Wikipedia:Huggle text/.. Huggle/2.1.19 url

soso

help.soso.com/webspider.htm text/.. Sosospider(url)

help.soso.com/webspider.htm - Sosospider(url)

discoveryengine

discoveryengine.com/discobot.html text/.. Mozilla/5.0 (compatible; discobot/2.0; url)

discoveryengine.com/discobot.html - Mozilla/5.0 (compatible; discobot/2.0; url)

discoveryengine.com/discobot.html image/.. Mozilla/5.0 (compatible; discobot/2.0; url)

yacy

yacy.net/bot.html text/.. yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url

yacy.net/bot.html text/.. yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url

yacy.net/bot.html text/.. yacybot (sciencenet-any; amd64 Linux 2.6.38-15-generic; java 1.6.0_22; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url

yacy.net/bot.html text/.. yacybot (freeworld-global; amd64 Linux 3.2.0; java 1.7.0_03; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-desktop; java 1.6.0_24; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.42.12-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url

yacy.net/bot.html text/.. yacybot (sciencenet-any; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.9; java 1.6.0_26; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Etc/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; i386 Linux 2.6.18-238.19.1.el5xen; java 1.6.0_22; America/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.0.30-vs2.3.2.3-dq67sw; java 1.6.0_24; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.0; java 1.7.0_03; Europe/en) url

yacy.net/bot.html - yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) url

yacy.net/bot.html text/.. yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Indian/en) url

yacy.net/bot.html - yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) url

yacy.net/bot.html - yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Etc/en) url

yacy.net/bot.html text/.. yacybot (freeworld/global; amd64 Linux 3.0.31-vs2.3.2.3-dq67sw; java 1.6.0_24; Europe/en) url

exabot

www.exabot.com/go/robot text/.. Mozilla/5.0 (compatible; Exabot/3.0; url)

www.exabot.com/go/robot - Mozilla/5.0 (compatible; Exabot/3.0; url)

youdao

www.youdao.com/help/webmaster/spider/ text/.. Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )

www.youdao.com/help/webmaster/spider/ image/.. Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)

www.youdao.com/help/webmaster/spider/ - Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )

www.youdao.com/help/webmaster/spider/ text/.. Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)

toolbar.youdao.com/ image/.. Youdao Toolbar (url)

zum

help.zum.com/inquiry text/.. ZumBot/1.0 (ZUM Search; url)

help.zum.com/inquiry image/.. ZumBot/1.0 (ZUM Search; url)

zeebox

www.zeebox.com application/json Zeebox (url)

dataparksearch

dataparksearch.org/bot text/.. DataparkSearch/4.54-26052011 (url)

dataparksearch.org/bot text/.. DataparkSearch/4.54-2012-04-08 (url)

wwwgogetpapers

wwwgogetpapers.com/ application/json User-Agent: GoGetPapersBot (url)

toolserver

wiki.toolserver.org/view/GeoHack text/.. Geohack (url)

toolserver.org/~dispenser/ text/.. DispensersTools (url)

toolserver.org/~para/cgi-bin/kmlexport text/.. url libwww-perl/6.02

toolserver.org/~dispenser/ application/json DispensersTools (url)

yioop

www.yioop.com/bot.php text/.. Mozilla/5.0 (compatible; YioopBot; url)

www.yioop.com/bot.php image/.. Mozilla/5.0 (compatible; YioopBot; url)

sistrix

crawler.sistrix.net/ text/.. Mozilla/5.0 (compatible; SISTRIX Crawler; url)

flipboard

flipboard.com/browserproxy image/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)

flipboard.com/browserproxy text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)

flipboard.com/browserproxy text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)

flipboard.com/browserproxy application/json Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)

flipboard.com/browserproxy - Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)

daum

tab.search.daum.net/aboutWebSearch.html text/.. Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0

FeedBurner

www.FeedBurner.com text/.. FeedBurner/1.0 (url)

wikidict

www.wikidict.de text/.. url

gnip

www.gnip.com/ text/.. UnwindFetchor/1.0 (url)

www.gnip.com/ image/.. UnwindFetchor/1.0 (url)

www.gnip.com/ - UnwindFetchor/1.0 (url)

traslated

mymemory.traslated.net/doc/ text/.. Mozilla/5.0 (MyMemory Bot url)

bin-co

www.bin-co.com/php/scripts/load/ text/.. BinGet/1.00.A (url)

www.bin-co.com/php/scripts/load/ application/vnd.php.serialized BinGet/1.00.A (url)

archive

www.archive.org/details/archive.org_bot text/.. Mozilla/5.0 (compatible; archive.org_bot url)

www.archive.org/details/archive.org_bot image/.. Mozilla/5.0 (compatible; archive.org_bot url)

archive.org/details/archive.org_bot image/.. Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)

www.archive.org image/.. Mozilla/5.0 (compatible; heritrix/3.1.0 url)

www.archive.org text/.. Mozilla/5.0 (compatible; heritrix/3.1.0 url)

www.archive.org/details/archive.org_bot text/.. Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)

sf

magpierss.sf.net text/.. MagpieRSS/0.7x (url)

liferea.sf.net/ text/.. Liferea/1.x.x (Linux; es_ES.UTF-8; url)

liferea.sf.net/ text/.. Liferea/0.x.x (Linux; en_US.UTF-8; url)

magpierss.sf.net text/.. MagpieRSS/0.72 (url; No cache)

tweetmeme

tweetmeme.com/ text/.. Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)

tweetmeme.com/ - Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)

commoncrawl

www.commoncrawl.org/bot.html text/.. CCBot/1.0 (url)

goo

help.goo.ne.jp/contact/ text/.. goo wikipedia (url)

help.goo.ne.jp/door/crawler.html text/.. ichiro/3.0 (url)

bnf

www.bnf.fr/fr/outils/a.dl_web_capture_robot.html text/.. Mozilla/5.0 (compatible; bnf.fr_bot; url)

www.bnf.fr/fr/outils/a.dl_web_capture_robot.html image/.. Mozilla/5.0 (compatible; bnf.fr_bot; url)

www.bnf.fr/fr/outils/a.dl_web_capture_robot.html - Mozilla/5.0 (compatible; bnf.fr_bot; url)

jike

shoulu.jike.com/spider.html text/.. Mozilla/5.0 (compatible; JikeSpider; url)

speaktoit

www.speaktoit.com application/json Speaktoit url

kosmix

www.kosmix.com/html/kosmos.html application/xml Mozilla/5.0(compatible;Kosmos/1.0;url)

ephorus

www.ephorus.com/ text/.. Mozilla/5.0 (compatible; Ephorusbot/1.3.0; url)

neofonie

spider.neofonie.de text/.. MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to mail address ; url; mail address )

spider.neofonie.de text/.. mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to mail address ; url; mail address )

enwp

enwp.org/User:SDPatrolBot text/.. SDPatrolBot (url)

enwp.org/User:KingpinBot text/.. KingpinBot (url)

enwp.org/User:H3llkn0wz/WikiSharpAPI text/.. WikiSharpAPI/0.3 url (C# .NET)

federatedmedia

federatedmedia.net text/.. Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1

archive-it

archive-it.org/files/site-owners.html image/.. Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)

archive-it.org/files/site-owners.html text/.. Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)

archive-it.org/files/site-owners.html - Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)

avantbrowser

www.avantbrowser.com text/.. Avant Browser (url)

www.avantbrowser.com text/.. Advanced Browser (url)

mediawiki

www.mediawiki.org/ text/.. MediaWiki OAI Harvester 0.2 (url)

www.mediawiki.org/ text/.. MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)

github

github.com/pauldix/typhoeus/tree/master text/.. Typhoeus - url

github.com/NeilCrosby/wikislurp application/vnd.php.serialized WikiSlurp (url)

github.com/edsu/wikitweets application/json wikitweets <url

kalooga

kalooga.com/crawler image/.. Mozilla/5.0 (compatible; KaloogaBot; url)

kalooga.com/crawler text/.. Mozilla/5.0 (compatible; KaloogaBot; url)

emining

emining.jp/ text/.. emBot-GalaBuzz/Nutch-1.0 (url; mail address )

butterflyfacts

www.butterflyfacts.net text/.. WordPress/3.3.2; url

www.butterflyfacts.net text/.. WordPress/3.1.2; url

jetbrains

www.jetbrains.com/omea_reader/ text/.. JetBrains Omea Reader 2.0 Release Candidate 1 (url)

www.jetbrains.com/omea_reader/ text/.. JetBrains Omea Reader 1.0.x (url)

feedshow

www.feedshow.com text/.. FeedshowOnline (url)

www.feedshow.com text/.. Feedshow/x.0 (url; 1 subscriber)

newsgator

www.newsgator.com text/.. NewsGatorOnline/2.0 (url; 1 subscribers)

www.newsgator.com/ text/.. FeedDemon/2.7 (url; Microsoft Windows XP)

wikimpress

wikimpress.org/ text/.. Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0

freebase

www.freebase.com text/.. metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)

apercite

www.apercite.fr/robot/index.html image/.. Mozilla/5.0 (compatible; Apercite; url)

xbmc

www.xbmc.org image/.. XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)

www.xbmc.org image/.. XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B206f); url)

www.xbmc.org image/.. XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)

www.xbmc.org image/.. XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1 (Build 9B179b); url)

netseer

www.netseer.com/crawler.html text/.. Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )

bibalex

archive.bibalex.org/bot/ image/.. Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)

archive.bibalex.org/bot/ text/.. Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)

paper

support.paper.li/entries/20023257-what-is-paper-li text/.. Mozilla/5.0 (compatible; PaperLiBot/2.1; url)

veveo

corporate.veveo.net/webmasters.html text/.. Mozilla/5.0 (compatible; Veveobot; url)

entireweb

www.entireweb.com/about/search_tech/speedy_spider/ text/.. Mozilla/5.0 (Windows; Windows NT 5.1; en-US) Speedy Spider (url)

hatena

a.hatena.ne.jp/help text/.. Hatena Antenna/0.5 (url)

mgw.hatena.ne.jp/help text/.. DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; url)

easybib

content.easybib.com/autocite/ text/.. EasyBib AutoCite (url)

content.easybib.com/autocite/ application/json EasyBib AutoCite (url)

edu:8080

vancouver.cs.washington.edu:8080/ text/.. Mozilla/5.0/heritrix/3.1.0 (compatible;; url)

abonti

www.abonti.com text/.. Mozilla/5.0 (compatible; Abonti/0.91 - url)

kr:6600

www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp text/.. url

whatrhymeswith

www.whatrhymeswith.com/site/rhyme-bot text/.. RhymeBot/0.1 (url)

textdigger

textdigger.com text/.. Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1

bsurprised

bsurprised.com/ text/.. BSurprised WikiBox 0.1.3 (url)

wikiglass

wikiglass.com text/.. url : mail address

tiscali

www.tiscali.it/ text/.. Mozilla/5.0 (compatible; IstellaBot/1.01.18 url)

SearchNearMe

SearchNearMe.com/contact.php text/.. SearchNearMe (url)

SearchNearMe.com/contact.php application/vnd.php.serialized SearchNearMe (url)

orcabrowser

www.orcabrowser.com text/.. Orca Browser (url)

it-influentials

search.it-influentials.com/bot.htm text/.. Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)

graemef

graemef.com text/.. NewsGator FetchLinks extension/0.2.0 (url)

feeds4all

www.feeds4all.com/feedzcollector text/.. FeedZcollector v1.x (Platinum) url

tinyurl

tinyurl.com/64t5n text/.. Rome Client (url) Ver: 0.9

zootycoon

www.zootycoon.com text/.. Zoo Tycoon 2 Client -- url

snarfware

www.snarfware.com/ text/.. Snarfer/0.x.x (url)

rssbandit

www.rssbandit.org text/.. RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )

superfeedr

superfeedr.com application/xml Superfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard

trendiction

www.trendiction.de/bot text/.. Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11

nemui

mozshot.nemui.org/ text/.. Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)

seebot

seebot.org text/.. Lynx/2.8 (;url)

kula

kula.jp/endo text/.. endo/1.0 (Mac OS X; ppc i386; url)

ponderer

ponderer.org/download/annotate_google.user.js text/.. annotate_google; url

netnewswireapp

netnewswireapp.com/mac/ - NetNewsWire/3.3 (Mac OS X; url; gzip-happy)

timewe

timewe.net text/.. CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0

zipcommander

www.zipcommander.com/ text/.. 1st ZipCommander (Net) - url

drupal

drupal.org/ text/.. User-Agent: Drupal (url)

drupal.org/ text/.. Drupal (url)

weblio

www.weblio.jp/ text/.. Mozilla/5.0 (compatible; WeblioBot; url)

proximic

www.proximic.com text/.. Mozilla/5.0 (compatible; proximic; url)

blogbridge

www.blogbridge.com/ text/.. BlogBridge 2.13 (url)

winpodder

winpodder.com text/.. WinPodder (url)

rssreader

www.rssreader.com text/.. RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0

nb

www.nb.no/vevfangst image/.. Mozilla/5.0 (compatible; heritrix/1.14.4 url)

www.nb.no/vevfangst text/.. Mozilla/5.0 (compatible; heritrix/1.14.4 url)

ranchero

ranchero.com/netnewswire/ text/.. NetNewsWire/2.x (Mac OS X; url)

warebay

www.warebay.com/bot.html text/.. Mozilla/5.0 (compatible; WBSearchBot/1.1; url)

plagger

plagger.org/ text/.. Plagger/0.x.xx (url)

semager

www.semager.de/blog/semager-bots/ text/.. Mozilla/5.0 (compatible; Semager/1.4c; url)

spotinfluence

spotinfluence.com text/.. spotinfluence/Nutch-1.4 (Spot Influence crawler; url; mail address )

spotinfluence.com - spotinfluence/Nutch-1.4 (Spot Influence crawler; url; mail address )

plagiarismcheck

plagiarismcheck.org application/json WikiCrawl 1.0b (url contact-mail: mail address )

cognarius

cognarius.com application/json AppsArlak/1.0 (url)

embed

support.embed.ly/ text/.. Mozilla/5.0 (compatible; Embedly/0.2; url)

support.embed.ly/ image/.. Mozilla/5.0 (compatible; Embedly/0.2; snap; url)

rockpeaks

www.rockpeaks.com/contact text/.. RockPeaks/0.1 (url)

apache

lucene.apache.org/nutch/bot.html text/.. NutchCVS/0.7.2 (Nutch; url; mail address )

simplepie

simplepie.org application/xml SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103

simplepie.org text/.. SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103

alexa

www.alexa.com/site/help/webmasters text/.. ia_archiver (url; mail address )

duckduckgo

duckduckgo.com/duckduckbot.html text/.. DuckDuckBot/1.1; (url)

duckduckgo.com/duckduckpreview.html - DuckDuckPreview/1.0; (url)

duckduckgo.com/duckduckpreview.html text/.. DuckDuckPreview/1.0; (url)

ac

cse.iitkgp.ac.in/~rpr text/.. Rajendra/Nutch-1.4 (Researcher; url; mail address )

www.clips.ua.ac.be/pages/pattern application/json Pattern/2.3 url

netvibes

www.netvibes.com text/.. Netvibes (url)

spinn3r

spinn3r.com/robot text/.. Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19

grid-son

grid-son.com application/json url

newmedialab

labs.newmedialab.at/iSignals/ text/.. LMF 2.0rc6-SNAPSHOT/LinkedDataClientService running at url

govid

govid.mobi/bot.php text/.. Mozilla/5.0 (compatible; gofind; url)

linkbutler

www.linkbutler.de/spider text/.. lb-spider/Mozilla/5.0 Gecko/20100101 Firefox/10.0.2 (lb-spider; url; mail address )

Anonymouse

Anonymouse.org/ image/.. url (Unix)

Anonymouse.org/ text/.. url (Unix)

whstour

whstour.com/osaka text/.. WordPress/3.3.1; url

whstour.com/tokyo text/.. WordPress/3.3.1; url

whstour.com/nagoya text/.. WordPress/3.3.1; url

topsy

labs.topsy.com/butterfly/ text/.. Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8

genevasearch

www.genevasearch.com/ text/.. gva/europe (Geneva Web Explorer.; url; mail address )

searchtechnologies

www.searchtechnologies.com text/.. Mozilla/5.0 (compatible; heritrix/1.14.3 url)

zapbot

www.zapbot.com text/.. Mozilla/5.0 (compatible; ZapBot/0.2c; url)

www.zapbot.net text/.. Mozilla/5.0 (compatible; ZapBot/0.2n; url)

www.zapbot.org text/.. Mozilla/5.0 (compatible; ZapBot/0.2o; url)

sonyericsson

www.sonyericsson.com/UAprof/R800xR301.xml image/.. Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1

www.sonyericsson.com/UAprof/R800xR301.xml text/.. Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1

instapaper

www.instapaper.com/ text/.. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)

pinterest

pinterest.com/ image/.. Pinterest/0.1 url

froute

labs.froute.jp/pc2m/help.html text/.. Froute Mobile Gateway/1.0 (url)

thearchangelmichael

ref.thearchangelmichael.net text/.. WordPress/3.3.1; url

thearchangelmichael.net text/.. WordPress/3.3.1; url

ibis

ibis.ne.jp/browser/about.html image/.. Mozilla/4.0 (compatible; ibisBrowser; url)

edu

ws.nju.edu.cn/falcons/ text/.. Mozilla/5.0 (compatible; Falconsbot; url)

suggy

blog.suggy.com/was-ist-suggy/suggy-webcrawler/ text/.. Mozilla/5.0 (compatible; suggybot v0.01a, url)

turnitin

www.turnitin.com/robot/crawlerinfo.html text/.. TurnitinBot/2.1 (url)

102392.059999993 total

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines. Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters). GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot For this report page requests are considered to be issued by a crawler in two cases: 1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated) 2 The user agent string contains the term bot, spider or crawl[er]' In total 67,897,030 page requests (mime type text/html only!) per day are considered crawler requests, out of 478,729,940 external requests, which is 14.2%
Page requests for crawlers that specify a url in the agent string
Count x 1000	Secondary domain (~site) name	URL	Mime type	User agent
	google
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		desktop.google.com/	text/..	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/feedfetcher.html	-	FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/json	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortografia4)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien4)
		www.google.com/feedfetcher.html	application/xml	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ortopedianew)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~cloudcrawling)
		code.google.com/p/crawler4j/	text/..	crawler4j (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wikien3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: rarplayer)
		www.google.com/feedfetcher.html	text/..	FeedFetcher-Google; (url)
		www.google.com/feedfetcher.html	application/xml	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: s~senchaiosrc)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: s~redconceptual)
		desktop.google.com/	-	Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
		code.google.com/appengine	application/xml	AppEngine-Google; (url; appid: wikipedia-raw)
		www.google.com/feedfetcher.html	-	Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pakgalaxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: myproxywx)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: abdulfat)
		www.google.com/coop/cse/cref	text/..	FeedFetcher-Google-CoOp; (url)
		code.google.com/appengine	text/..	WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; documents; url)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kires-roxy)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/feedfetcher.html	text/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~wikigraph2)
		code.google.com/appengine	image/..	Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: bie99miracle)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxy-devakishor)
		code.google.com/appengine	text/..	www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
		code.google.com/appengine	application/json	MWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki4)
		www.google.com/feedfetcher.html	image/..	Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~deutiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~drizzlprox)
		docs.google.com	image/..	Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url)
		code.google.com/appengine	image/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: atxproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nation4india)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vi-mobile)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tusawebproxy4)
		code.google.com/p/ronda	application/json	Ronda - url
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlp6)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: d24-img)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: worldwide-propaganda)
		desktop.google.com/	application/xml	Mozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlp9)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: free-data)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nagarajhubli-proxy-server)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline0)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: 114proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: chris-homework-helper)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline9)
		docs.google.com	text/..	Mozilla/5.0 (compatible; GoogleDocs; conversion; url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: simple-tools2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: vebproxy)
		code.google.com/appengine	-	AppEngine-Google; (url; appid: s~senchaiosrc)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~zagrobelnyprox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: proxyusing121)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: maltingsproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pazvantoff)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~harunakaze)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dkoxyserv)
		desktop.google.com/	image/..	Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
		code.google.com/appengine	text/..	Mozilla/5.0 AppEngine-Google; (url; appid: s~app3123ak)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlp8)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: paradigm-web-proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~kyaysarlay)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: adrianswebproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~proxyseekkety)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: betafxserver)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web-proxy-hh)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: pox)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: weps002)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wwwwebp0)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tmobile-internet)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: 9329269)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: hideproxyz)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: yuricamara)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: 9000tunnels)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: taterproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wwwwebp2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webproxy8-2)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: usawebdl3)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ivegotalovelybunch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: simple-tools6)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~crowdsurfer100)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: thetechnolust)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: jptaravellahighschool)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nhsportal)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: gaucho-labnol)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: python-proxy-server)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: nothing-unusual)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ideserveinternet)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: misc-tools)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wiwohk-proxy-server)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wagagate)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~francetiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ieultimateproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~keytanwiki1)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kbworld24)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: davidgotmoney50)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: openeyeproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: threewiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webproxy8-9)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: azamasmadi)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: predictionwrong)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ageryder)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: sony-words)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kaveriselvaraj)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: sizzsurf)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: mboharsik)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: dex-dwds)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web4proxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: ivankrisproxyserver)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kuryproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: burmansearch)
		www.google.com/feedfetcher.html	image/..	FeedFetcher-Google; (url)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~misterhac)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: downarchivestuffproxyserver)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: cachehew)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: discretepword)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~link123451)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webproxy8-5)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~adddon1)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web-mastered)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: quigonjinn03)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~japantiki)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: boxapp)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: kerouanen)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: laaplicaciondelucas)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~tyangbanga)
		code.google.com/appengine	text/..	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline5)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: wwwwebp8)
		code.google.com/appengine	application/json	AppEngine-Google; (url; appid: prfleme)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webusadlq0)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: webponline8)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: seiyukyouen)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~nyinayminproxy2)
		www.google.com/bot.html	NONE/wikipedia	- Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		code.google.com/appengine	text/..	oohEmbed.com AppEngine-Google; (url; appid: vipoembed)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: djshan45)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tgbeeson)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: mehproxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: azzaziprxy)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: tinkernutsearch)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: s~sony-hack)
		code.google.com/appengine	text/..	Wiki.java 0.25 AppEngine-Google; (url; appid: wikipediatools)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: trabserver)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: futureducation1)
		code.google.com/appengine	text/..	AppEngine-Google; (url; appid: web-phpproxy)
	facebook
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.0 (url)
		www.facebook.com/externalhit_uatext.php	text/..	facebookexternalhit/1.1 (url)
		developers.facebook.com	image/..	facebookplatform/1.0 (url)
		developers.facebook.com	-	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	image/..	facebookexternalhit/1.1 (url)
		developers.facebook.com	text/..	facebookplatform/1.0 (url)
		www.facebook.com/externalhit_uatext.php	-	facebookexternalhit/1.1 (url)
		www.facebook.com/externalhit_uatext.php	application/vnd.php.serialized	facebookexternalhit/1.1 (url)
	bing
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	-	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	application/vnd.php.serialized	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
		www.bing.com/bingbot.htm	image/..	Mozilla/5.0 (compatible; bingbot/2.0; url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html))
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: images-jpg)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) (via babelfish.yahoo.com)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible;bingbot/2.0;url)
		www.bing.com/bingbot.htm	text/..	Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: p8roxy)
	google?
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	GoogleBot/2.1 (url)
		www.google.com/bot.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	image/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	-	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	-	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	application/xml	Mozilla/5.0 (compatible; GoogleBot/2.1; url)
		www.google.com/bot.html	text/..	Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
		www.google.com/bot.html	text/..	SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
	baidu
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider-image(url)
		www.baidu.com/search/spider.html	-	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	image/..	Baiduspider-image(url)
		www.baidu.com/search/spider.html	image/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	application/vnd.php.serialized	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.html	text/..	Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: misc-tools)
		www.baidu.com/search/spider.html	application/ogg	Mozilla/5.0 (compatible; Baiduspider/2.0; url)
		www.baidu.com/search/spider.htm	-	Baiduspider-image(url)
		www.baidu.com/search/spider.htm	text/..	Baiduspider(url)
	yandex
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexDirect/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImages/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
		yandex.com/bots	application/vnd.php.serialized	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	image/..	Mozilla/5.0 (compatible; YandexBot/3.0; url)
		yandex.com/bots	text/..	Mozilla/5.0 (compatible; YandexNewslinks; url)
		yandex.com/bots	-	Mozilla/5.0 (compatible; YandexAntivirus/2.0; url)
	naver
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	-	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	image/..	Yeti/1.0 (NHN Corp.; url)
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
		help.naver.com/robots/	text/..	Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
		corp.naver.jp/	text/..	Mozilla/5.0 (compatible; NaverJapan/1.0; url)
	yahoo
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		help.yahoo.com/help/us/ysearch/slurp	image/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
		listing.yahoo.co.jp/support/faq/int/other/other_001.html	text/..	Y!J-BRJ/YATS crawler (url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	-	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.com/help/us/ysearch/slurp	-	Mozilla/5.0 (compatible; Yahoo! Slurp; url)
		developer.yahoo.com/yql/provider	text/..	Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	image/..	'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
		help.yahoo.com/help/us/ysearch/slurp	application/vnd.php.serialized	Mozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
		help.yahoo.com/help/us/ysearch/slurp	text/..	Mozilla/5.0 (compatible; Yahoo! Nano; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRI/0.0.1 crawler ( url )
		help.yahoo.com	text/..	Mozilla/5.0 (YahooYSMcm/3.0.0; url)
		help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html	text/..	Y!J-BRT/1.0 crawler (url)
		help.yahoo.com/help/us/ysearch/slurp	application/json	Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
	80legs
		www.80legs.com/webcrawler.html	text/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	-	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
		www.80legs.com/webcrawler.html	image/..	Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
	msn
		search.msn.com/msnbot.htm	text/..	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-Products/1.0 (url)
		search.msn.com/msnbot.htm	image/..	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	text/..	msnbot-UDiscovery/2.0b (url)
		search.msn.com/msnbot.htm	text/..	msnbot-NewsBlogs/2.0b (url)
		search.msn.com/msnbot.htm	-	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	-	msnbot/2.0b (url)
		search.msn.com/msnbot.htm	application/vnd.php.serialized	msnbot-media/1.1 (url)
		search.msn.com/msnbot.htm	-	msnbot/0.01 (url)
	blekko
		blekko.com/about/blekkobot	text/..	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
		blekko.com/about/blekkobot	-	Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
	sblog
		fulltext.sblog.cz/screenshot/	image/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	text/..	SeznamBot/3.0 (url)
		fulltext.sblog.cz/screenshot/	text/..	Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
		fulltext.sblog.cz/	-	SeznamBot/3.0 (url)
	cibra
		cibra.de/	text/..	CiBra Data Collector (url)
	ahrefs
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
		ahrefs.com/robot/	-	Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
		ahrefs.com/robot/	text/..	Mozilla/5.0 (compatible; AhrefsBot/2.0; url)
		ahrefs.com/robot/	application/ogg	Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
	www.
		www.	text/..	GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
		www.	text/..	GoogleBot/2.1 (urlGoogleBot.com/bot.html)
	php
		pear.php.net/	application/vnd.php.serialized	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/0.5.2 (url) PHP/5.2.17
		pear.php.net/	text/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	application/xml	HTTP_Request2/2.0.0 (url) PHP/5.3.8
		pear.php.net/	image/..	PEAR HTTP_Request class ( url )
		pear.php.net/package/http_request2	text/..	HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.14
		pear.php.net/	application/xml	PEAR HTTP_Request class ( url )
	sogou
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	-	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	image/..	Sogou Pic Spider/3.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	application/vnd.php.serialized	Sogou web spider/4.0(url)
		www.sogou.com/docs/help/webmasters.htm#07	text/..	Sogou Pic Spider/3.0(url)
	majestic12
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
		www.majestic12.co.uk/bot.php?	text/..	Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
	echonest
		the.echonest.com/reader/	application/xml	nestReader/0.3 (discovery; url; reader at echonest.com)
		the.echonest.com/reader/	text/..	nestReader/0.3 (discovery; url; reader at echonest.com)
	wordpress
		klausgauger.wordpress.com	text/..	WordPress/3.4-beta4-20825; url
		klima47.wordpress.com	text/..	WordPress/3.4-beta4-20725; url
		kingcrimsonprog.wordpress.com	image/..	WordPress/3.4-beta4-20766; url
	wikipedia
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.0 url
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WikiCleaner (url)
		en.wikipedia.org/wiki/User_talk:Blevintron	text/..	BlevintronBot version 2012-05-19 contact url
		en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation	text/..	WPCleaner (url)
		en.wikipedia.org/wiki/User_talk:Blevintron	text/..	BlevintronBot version 2012-05-16 contact url
		en.wikipedia.org/wiki/User_talk:Blevintron	text/..	BlevintronBot version 2012-05-09 contact url
		fr.wikipedia.org/wiki/Utilisateur:Salebot	application/json	Salebot, see url (uses Perl MediaWiki::API)
		sk.wikipedia.org/wiki/Redaktor:TeslaBot	-	TeslaBot (url)
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.18.0 url
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19.2 url
		en.wikipedia.org/wiki/Wikipedia:Huggle	text/..	Huggle/2.1.19 url
	soso
		help.soso.com/webspider.htm	text/..	Sosospider(url)
		help.soso.com/webspider.htm	-	Sosospider(url)
	discoveryengine
		discoveryengine.com/discobot.html	text/..	Mozilla/5.0 (compatible; discobot/2.0; url)
		discoveryengine.com/discobot.html	-	Mozilla/5.0 (compatible; discobot/2.0; url)
		discoveryengine.com/discobot.html	image/..	Mozilla/5.0 (compatible; discobot/2.0; url)
	yacy
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.32-33-generic; java 1.6.0_20; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.38-15-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 3.2.0; java 1.7.0_03; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-desktop; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.42.12-1.fc15.x86_64; java 1.6.0_22; W-SU/ru) url
		yacy.net/bot.html	text/..	yacybot (sciencenet-any; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.9; java 1.6.0_26; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; i386 Linux 2.6.18-238.19.1.el5xen; java 1.6.0_22; America/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.30-vs2.3.2.3-dq67sw; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0; java 1.7.0_03; Europe/en) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Indian/en) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) url
		yacy.net/bot.html	-	yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Etc/en) url
		yacy.net/bot.html	text/..	yacybot (freeworld/global; amd64 Linux 3.0.31-vs2.3.2.3-dq67sw; java 1.6.0_24; Europe/en) url
	exabot
		www.exabot.com/go/robot	text/..	Mozilla/5.0 (compatible; Exabot/3.0; url)
		www.exabot.com/go/robot	-	Mozilla/5.0 (compatible; Exabot/3.0; url)
	youdao
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	image/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		www.youdao.com/help/webmaster/spider/	-	Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
		www.youdao.com/help/webmaster/spider/	text/..	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
		toolbar.youdao.com/	image/..	Youdao Toolbar (url)
	zum
		help.zum.com/inquiry	text/..	ZumBot/1.0 (ZUM Search; url)
		help.zum.com/inquiry	image/..	ZumBot/1.0 (ZUM Search; url)
	zeebox
		www.zeebox.com	application/json	Zeebox (url)
	dataparksearch
		dataparksearch.org/bot	text/..	DataparkSearch/4.54-26052011 (url)
		dataparksearch.org/bot	text/..	DataparkSearch/4.54-2012-04-08 (url)
	wwwgogetpapers
		wwwgogetpapers.com/	application/json	User-Agent: GoGetPapersBot (url)
	toolserver
		wiki.toolserver.org/view/GeoHack	text/..	Geohack (url)
		toolserver.org/~dispenser/	text/..	DispensersTools (url)
		toolserver.org/~para/cgi-bin/kmlexport	text/..	url libwww-perl/6.02
		toolserver.org/~dispenser/	application/json	DispensersTools (url)
	yioop
		www.yioop.com/bot.php	text/..	Mozilla/5.0 (compatible; YioopBot; url)
		www.yioop.com/bot.php	image/..	Mozilla/5.0 (compatible; YioopBot; url)
	sistrix
		crawler.sistrix.net/	text/..	Mozilla/5.0 (compatible; SISTRIX Crawler; url)
	flipboard
		flipboard.com/browserproxy	image/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
		flipboard.com/browserproxy	text/..	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
		flipboard.com/browserproxy	application/json	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
		flipboard.com/browserproxy	-	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
	daum
		tab.search.daum.net/aboutWebSearch.html	text/..	Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
	FeedBurner
		www.FeedBurner.com	text/..	FeedBurner/1.0 (url)
	wikidict
		www.wikidict.de	text/..	url
	gnip
		www.gnip.com/	text/..	UnwindFetchor/1.0 (url)
		www.gnip.com/	image/..	UnwindFetchor/1.0 (url)
		www.gnip.com/	-	UnwindFetchor/1.0 (url)
	traslated
		mymemory.traslated.net/doc/	text/..	Mozilla/5.0 (MyMemory Bot url)
	bin-co
		www.bin-co.com/php/scripts/load/	text/..	BinGet/1.00.A (url)
		www.bin-co.com/php/scripts/load/	application/vnd.php.serialized	BinGet/1.00.A (url)
	archive
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; archive.org_bot url)
		www.archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; archive.org_bot url)
		archive.org/details/archive.org_bot	image/..	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
		www.archive.org	image/..	Mozilla/5.0 (compatible; heritrix/3.1.0 url)
		www.archive.org	text/..	Mozilla/5.0 (compatible; heritrix/3.1.0 url)
		www.archive.org/details/archive.org_bot	text/..	Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 url)
	sf
		magpierss.sf.net	text/..	MagpieRSS/0.7x (url)
		liferea.sf.net/	text/..	Liferea/1.x.x (Linux; es_ES.UTF-8; url)
		liferea.sf.net/	text/..	Liferea/0.x.x (Linux; en_US.UTF-8; url)
		magpierss.sf.net	text/..	MagpieRSS/0.72 (url; No cache)
	tweetmeme
		tweetmeme.com/	text/..	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
		tweetmeme.com/	-	Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
	commoncrawl
		www.commoncrawl.org/bot.html	text/..	CCBot/1.0 (url)
	goo
		help.goo.ne.jp/contact/	text/..	goo wikipedia (url)
		help.goo.ne.jp/door/crawler.html	text/..	ichiro/3.0 (url)
	bnf
		www.bnf.fr/fr/outils/a.dl_web_capture_robot.html	text/..	Mozilla/5.0 (compatible; bnf.fr_bot; url)
		www.bnf.fr/fr/outils/a.dl_web_capture_robot.html	image/..	Mozilla/5.0 (compatible; bnf.fr_bot; url)
		www.bnf.fr/fr/outils/a.dl_web_capture_robot.html	-	Mozilla/5.0 (compatible; bnf.fr_bot; url)
	jike
		shoulu.jike.com/spider.html	text/..	Mozilla/5.0 (compatible; JikeSpider; url)
	speaktoit
		www.speaktoit.com	application/json	Speaktoit url
	kosmix
		www.kosmix.com/html/kosmos.html	application/xml	Mozilla/5.0(compatible;Kosmos/1.0;url)
	ephorus
		www.ephorus.com/	text/..	Mozilla/5.0 (compatible; Ephorusbot/1.3.0; url)
	neofonie
		spider.neofonie.de	text/..	MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to mail address ; url; mail address )
		spider.neofonie.de	text/..	mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to mail address ; url; mail address )
	enwp
		enwp.org/User:SDPatrolBot	text/..	SDPatrolBot (url)
		enwp.org/User:KingpinBot	text/..	KingpinBot (url)
		enwp.org/User:H3llkn0wz/WikiSharpAPI	text/..	WikiSharpAPI/0.3 url (C# .NET)
	federatedmedia
		federatedmedia.net	text/..	Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
	archive-it
		archive-it.org/files/site-owners.html	image/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
		archive-it.org/files/site-owners.html	text/..	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
		archive-it.org/files/site-owners.html	-	Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
	avantbrowser
		www.avantbrowser.com	text/..	Avant Browser (url)
		www.avantbrowser.com	text/..	Advanced Browser (url)
	mediawiki
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url)
		www.mediawiki.org/	text/..	MediaWiki OAI Harvester 0.2 (url) (client id: nttr.co.jp; experimental)
	github
		github.com/pauldix/typhoeus/tree/master	text/..	Typhoeus - url
		github.com/NeilCrosby/wikislurp	application/vnd.php.serialized	WikiSlurp (url)
		github.com/edsu/wikitweets	application/json	wikitweets <url
	kalooga
		kalooga.com/crawler	image/..	Mozilla/5.0 (compatible; KaloogaBot; url)
		kalooga.com/crawler	text/..	Mozilla/5.0 (compatible; KaloogaBot; url)
	emining
		emining.jp/	text/..	emBot-GalaBuzz/Nutch-1.0 (url; mail address )
	butterflyfacts
		www.butterflyfacts.net	text/..	WordPress/3.3.2; url
		www.butterflyfacts.net	text/..	WordPress/3.1.2; url
	jetbrains
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 2.0 Release Candidate 1 (url)
		www.jetbrains.com/omea_reader/	text/..	JetBrains Omea Reader 1.0.x (url)
	feedshow
		www.feedshow.com	text/..	FeedshowOnline (url)
		www.feedshow.com	text/..	Feedshow/x.0 (url; 1 subscriber)
	newsgator
		www.newsgator.com	text/..	NewsGatorOnline/2.0 (url; 1 subscribers)
		www.newsgator.com/	text/..	FeedDemon/2.7 (url; Microsoft Windows XP)
	wikimpress
		wikimpress.org/	text/..	Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
	freebase
		www.freebase.com	text/..	metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
	apercite
		www.apercite.fr/robot/index.html	image/..	Mozilla/5.0 (compatible; Apercite; url)
	xbmc
		www.xbmc.org	image/..	XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)

Page requests for probable crawlers, recognized by keyword

Count
x 1000 Agent string

Mime type (count ≥ 3)

PythonWikipediaBot/1.0

application/json

application/xml

text/..

-

application/x-www-form-urlencoded

image/..

php wikibot classes

application/vnd.php.serialized

text/..

GoogleBot-Image/1.0

text/..

image/..

-

MediaWikiCrawler-Google/2.0 ( mail address )

text/..

-

MoovidaBot/0.1

text/..

LinkParser/2.0

text/..

-

gsa-crawler (Enterprise; T2-DS3YYS6PYJWAS; mail address )

text/..

-

image/..

application/pdf

Peachy MediaWiki Bot API Version 1.0

application/vnd.php.serialized

text/..

GoogleBot-Image/1.0

text/..

image/..

-

application/vnd.php.serialized

application/json

Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )

text/..

-

wikiwix-bot-3.0

text/..

-

spider

text/..

application/vnd.php.serialized

application/json

image/..

Pywikipediabot/2.0

application/json

text/..

Answersbot

text/..

ClueBot/1.1

application/vnd.php.serialized

text/..

ClueBot/2.0

application/vnd.php.serialized

text/..

Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)

text/..

MediaWiki::Bot/1.00

text/..

application/json

-

BritannicaProjBot mail address

text/..

wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0

image/..

application/json

text/..

-

Mozilla 5.0 (Apibot 0.32)

application/vnd.php.serialized

python-wikitools/1.2 (User:BernsteinBot)

application/json

application/x-www-form-urlencoded

Mozilla/5.0 (compatible; Ezooms/1.0; mail address )

text/..

image/..

-

DigitalsmithsBot

text/..

DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )

text/..

application/xml

image/..

application/ogg

mail address

application/vnd.php.serialized

application/json

text/..

-

MediaWiki::Bot/3.2.6

application/json

text/..

mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)

application/x-www-form-urlencoded

application/json

Spider website 0.2

text/..

FAST Enterprise Crawler/5.3.4 ( mail address )

text/..

-

image/..

application/rsd+xml

AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])

application/json

text/..

GoogleBot 2.1

text/..

cis455crawler

text/..

-

GSLFbot

text/..

image/..

application/xml

DotNetWikiBot/2.100 (Unix 2.6.32.38; )

text/..

application/xml

plantspedia data crawler

text/..

Tawbot (public svn release; plwiki)

text/..

Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)

image/..

text/..

application/json

-

MLBot (www.metadatalabs.com/mlbot)

text/..

application/vnd.php.serialized

image/..

SineBot/1.5.18(User:SineBot)

application/vnd.php.serialized

text/..

DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )

text/..

application/x-www-form-urlencoded

application/xml

-

Wikibot/1.57 CFNetwork/520.4.3 Darwin/11.4.0 (x86_64) (MacBookPro8,2)

image/..

text/..

application/json

cleaner-wikipedia bot / self.maluke.com

text/..

application/json

DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )

text/..

application/xml

DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )

text/..

application/xml

-

AnomieBOT 1.0 (TagDater)

application/json

text/..

Bot work. [[no:User:PladaskBot]].

application/vnd.php.serialized

AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])

application/json

Test Webbot

text/..

-

Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )

text/..

application/opensearchdescription+xml

Webwiki Search Engine Bot - www.webwiki.de

text/..

php WalkingSoulBot

application/vnd.php.serialized

Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot

text/..

UCMore Crawler App

text/..

Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9

text/..

DotNetWikiBot/2.100 (Unix 5.10.0.0; )

text/..

application/xml

CorenSearchBot/1.5 en libwww-perl/6.02

text/..

AniBot/0.9 php/curl

application/vnd.php.serialized

AnomieBOT 1.0 (OrphanReferenceFixer)

application/json

LinksCrawler 0.1beta

text/..

-

image/..

Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1

image/..

text/..

-

application/ogg

DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )

text/..

-

SiocWikiBot/1.0

application/vnd.php.serialized

text/..

HTMLParser/1.6

text/..

application/x-wiki

image/..

SchoolReviewNetworkWikiBot

application/json

wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0

image/..

application/json

text/..

-

GermCrawler

application/json

text/..

Twitterbot/1.0

text/..

image/..

-

JavaCrawler/1.1

text/..

COIBot/1.00

text/..

FAST Enterprise Crawler 6 used by Wipro Ltd ( mail address )

text/..

-

AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])

application/json

SurakWare MediaWiki Bot/1.0

text/..

application/xml

DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )

text/..

TrueKnowledgeBot bot mail address >

application/vnd.php.serialized

application/xml

AdMedia bot

text/..

TVersity Media Robot

text/..

HTMLParser/2.0

text/..

image/..

OrlodrimBot/1.0

text/..

XLinkBot/1.00

text/..

GttBot/0.3

application/json

~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )

text/..

AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])

application/json

text/..

HosiryuhosiBot IRC-RecentChanges Util

-

text/..

bitlybot

text/..

image/..

-

AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])

application/json

super cool bot

application/vnd.php.serialized

SiteSeekerCrawler/1.0

text/..

HRoestBot, de-wikipedia using pywikipedia framework

text/..

application/json

Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9

image/..

text/..

-

Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1

image/..

text/..

-

MyCuteBot/0.1

text/..

application/json

application/vnd.php.serialized

AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])

application/json

text/..

AnomieBOT 1.0 (PERTableUpdater)

application/json

text/..

Mozilla/4.0 (compatible; merlinkbot/1.0)

text/..

uw_transparancy, toolserver.org wikiproject parser

application/json

AnomieBOT 1.0 (FlagIconRemover)

application/json

Applied-Technologies-Inc-Spider/Nutch-1.4

text/..

Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; mail address )

text/..

image/..

YBot/0.1

application/vnd.php.serialized

OrangeCrawler/Nutch-1.0 ( mail address )

text/..

wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0

image/..

application/json

text/..

-

DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )

text/..

application/xml

Tweakker crawler/Nutch-1.4

text/..

typedef web-crawler ( mail address ) for Yandex Data Analysis School

text/..

TheKeens bot

text/..

AnomieBOT 1.0 (TemplateSubster)

application/json

DotNetWikiBot/2.100 (Unix 3.0.0.12; )

text/..

application/xml

COIBot/2.0

text/..

SearchBot

text/..

FAST Enterprise Crawler 6 used by LexisNexis ( mail address )

text/..

-

image/..

Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.0/1.8.5.168;) Opera Mini/3.1

image/..

text/..

-

DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )

text/..

application/xml

Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.4.152;) Opera Mini/3.1

image/..

text/..

-

Goalkeeperbot(User:Beetstra)/1.0

text/..

Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 368 subscribers; feed-id=3852576738117026533)

application/xml

-

Opera/8.01 (J2ME/MIDP; MXit WebBot/6.1.0/1.8.5.168;) Opera Mini/3.1

image/..

text/..

-

python-wikitools/1.2 (User:LaraBot)

application/json

ClueBot/2.0 (ClueBot NG Report Interface)

text/..

My Nutch Spider/Nutch-1.4

text/..

AnomieBOT 1.0 (BAGBot)

application/json

text/..

HBC Archive Indexerbot 0.9a

text/..

MediaWiki::Bot 3.1.5

application/json

Baiduspider

text/..

WikiBot/0.1

text/..

Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)

application/xml

text/..

Mozilla 5.0 (Apibot 0.30b5)

application/vnd.php.serialized

Freebase Deathbot

text/..

EarwigBot/0.1.dev.gitbc9fdf28 (Python/2.7.1; https://github.com/earwig/earwigbot; mail address )

application/json

application/x-www-form-urlencoded

Empedia Bot

text/..

wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0

image/..

application/json

text/..

-

GNAA-bot

text/..

Mozilla/5.0 (Bgbot 0.5)

text/..

Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )"

application/xml

text/..

GoogleBot

text/..

image/..

FAST Enterprise Crawler/6.7.8 ( mail address )

text/..

-

Geni ircpybot 1.0

text/..

application/json

application/xml

infraEnterprise v8 Web Crawler

-

text/..

DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; )

text/..

UniversalFeedParser/5.1.1 https://code.google.com/p/feedparser/

text/..

application/xml

-

MediaWiki::Bot/v3.4.2

application/json

AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])

application/json

Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)

text/..

-

Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.4.152;) Opera Mini/3.1

image/..

text/..

-

OpenSearchServer_Bot

text/..

My Bot

text/..

image/..

Shad robot/1.0

text/..

MetallmanulBot for Wiktionary (run by Metallmanul)

application/json

19527.15 total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

Generated on Fri, Aug 10, 2012 12:10
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers

Page requests for probable crawlers, recognized by keyword
Count x 1000	Agent string
		Mime type (count ≥ 3)
	PythonWikipediaBot/1.0
		application/json
		application/xml
		text/..
		-
		application/x-www-form-urlencoded
		image/..
	php wikibot classes
		application/vnd.php.serialized
		text/..
	GoogleBot-Image/1.0
		text/..
		image/..
		-
	MediaWikiCrawler-Google/2.0 ( mail address )
		text/..
		-
	MoovidaBot/0.1
		text/..
	LinkParser/2.0
		text/..
		-
	gsa-crawler (Enterprise; T2-DS3YYS6PYJWAS; mail address )
		text/..
		-
		image/..
		application/pdf
	Peachy MediaWiki Bot API Version 1.0
		application/vnd.php.serialized
		text/..
	GoogleBot-Image/1.0
		text/..
		image/..
		-
		application/vnd.php.serialized
		application/json
	Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
		text/..
		-
	wikiwix-bot-3.0
		text/..
		-
	spider
		text/..
		application/vnd.php.serialized
		application/json
		image/..
	Pywikipediabot/2.0
		application/json
		text/..
	Answersbot
		text/..
	ClueBot/1.1
		application/vnd.php.serialized
		text/..
	ClueBot/2.0
		application/vnd.php.serialized
		text/..
	Mozilla/5.0 MaboMwFramework/1.1 (w:de:MerlIwBot)
		text/..
	MediaWiki::Bot/1.00
		text/..
		application/json
		-
	BritannicaProjBot mail address
		text/..
	wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	Mozilla 5.0 (Apibot 0.32)
		application/vnd.php.serialized
	python-wikitools/1.2 (User:BernsteinBot)
		application/json
		application/x-www-form-urlencoded
	Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
		text/..
		image/..
		-
	DigitalsmithsBot
		text/..
	DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		image/..
		application/ogg
	mail address
		application/vnd.php.serialized
		application/json
		text/..
		-
	MediaWiki::Bot/3.2.6
		application/json
		text/..
	mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
		application/x-www-form-urlencoded
		application/json
	Spider website 0.2
		text/..
	FAST Enterprise Crawler/5.3.4 ( mail address )
		text/..
		-
		image/..
		application/rsd+xml
	AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
		application/json
		text/..
	GoogleBot 2.1
		text/..
	cis455crawler
		text/..
		-
	GSLFbot
		text/..
		image/..
		application/xml
	DotNetWikiBot/2.100 (Unix 2.6.32.38; )
		text/..
		application/xml
	plantspedia data crawler
		text/..
	Tawbot (public svn release; plwiki)
		text/..
	Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
		image/..
		text/..
		application/json
		-
	MLBot (www.metadatalabs.com/mlbot)
		text/..
		application/vnd.php.serialized
		image/..
	SineBot/1.5.18(User:SineBot)
		application/vnd.php.serialized
		text/..
	DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/x-www-form-urlencoded
		application/xml
		-
	Wikibot/1.57 CFNetwork/520.4.3 Darwin/11.4.0 (x86_64) (MacBookPro8,2)
		image/..
		text/..
		application/json
	cleaner-wikipedia bot / self.maluke.com
		text/..
		application/json
	DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		application/xml
		-
	AnomieBOT 1.0 (TagDater)
		application/json
		text/..
	Bot work. [[no:User:PladaskBot]].
		application/vnd.php.serialized
	AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
		application/json
	Test Webbot
		text/..
		-
	Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
		text/..
		application/opensearchdescription+xml
	Webwiki Search Engine Bot - www.webwiki.de
		text/..
	php WalkingSoulBot
		application/vnd.php.serialized
	Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
		text/..
	UCMore Crawler App
		text/..
	Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
		text/..
	DotNetWikiBot/2.100 (Unix 5.10.0.0; )
		text/..
		application/xml
	CorenSearchBot/1.5 en libwww-perl/6.02
		text/..
	AniBot/0.9 php/curl
		application/vnd.php.serialized
	AnomieBOT 1.0 (OrphanReferenceFixer)
		application/json
	LinksCrawler 0.1beta
		text/..
		-
		image/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
		application/ogg
	DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
		text/..
		-
	SiocWikiBot/1.0
		application/vnd.php.serialized
		text/..
	HTMLParser/1.6
		text/..
		application/x-wiki
		image/..
	SchoolReviewNetworkWikiBot
		application/json
	wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	GermCrawler
		application/json
		text/..
	Twitterbot/1.0
		text/..
		image/..
		-
	JavaCrawler/1.1
		text/..
	COIBot/1.00
		text/..
	FAST Enterprise Crawler 6 used by Wipro Ltd ( mail address )
		text/..
		-
	AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
		application/json
	SurakWare MediaWiki Bot/1.0
		text/..
		application/xml
	DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
	TrueKnowledgeBot bot mail address >
		application/vnd.php.serialized
		application/xml
	AdMedia bot
		text/..
	TVersity Media Robot
		text/..
	HTMLParser/2.0
		text/..
		image/..
	OrlodrimBot/1.0
		text/..
	XLinkBot/1.00
		text/..
	GttBot/0.3
		application/json
	~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
		text/..
	AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
		application/json
		text/..
	HosiryuhosiBot IRC-RecentChanges Util
		-
		text/..
	bitlybot
		text/..
		image/..
		-
	AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
		application/json
	super cool bot
		application/vnd.php.serialized
	SiteSeekerCrawler/1.0
		text/..
	HRoestBot, de-wikipedia using pywikipedia framework
		text/..
		application/json
	Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
		image/..
		text/..
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	MyCuteBot/0.1
		text/..
		application/json
		application/vnd.php.serialized
	AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
		application/json
		text/..
	AnomieBOT 1.0 (PERTableUpdater)
		application/json
		text/..
	Mozilla/4.0 (compatible; merlinkbot/1.0)
		text/..
	uw_transparancy, toolserver.org wikiproject parser
		application/json
	AnomieBOT 1.0 (FlagIconRemover)
		application/json
	Applied-Technologies-Inc-Spider/Nutch-1.4
		text/..
	Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; mail address )
		text/..
		image/..
	YBot/0.1
		application/vnd.php.serialized
	OrangeCrawler/Nutch-1.0 ( mail address )
		text/..
	wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
		text/..
		application/xml
	Tweakker crawler/Nutch-1.4
		text/..
	typedef web-crawler ( mail address ) for Yandex Data Analysis School
		text/..
	TheKeens bot
		text/..
	AnomieBOT 1.0 (TemplateSubster)
		application/json
	DotNetWikiBot/2.100 (Unix 3.0.0.12; )
		text/..
		application/xml
	COIBot/2.0
		text/..
	SearchBot
		text/..
	FAST Enterprise Crawler 6 used by LexisNexis ( mail address )
		text/..
		-
		image/..
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.0/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
		text/..
		application/xml
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.4.152;) Opera Mini/3.1
		image/..
		text/..
		-
	Goalkeeperbot(User:Beetstra)/1.0
		text/..
	Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 368 subscribers; feed-id=3852576738117026533)
		application/xml
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/6.1.0/1.8.5.168;) Opera Mini/3.1
		image/..
		text/..
		-
	python-wikitools/1.2 (User:LaraBot)
		application/json
	ClueBot/2.0 (ClueBot NG Report Interface)
		text/..
	My Nutch Spider/Nutch-1.4
		text/..
	AnomieBOT 1.0 (BAGBot)
		application/json
		text/..
	HBC Archive Indexerbot 0.9a
		text/..
	MediaWiki::Bot 3.1.5
		application/json
	Baiduspider
		text/..
	WikiBot/0.1
		text/..
	Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de)
		application/xml
		text/..
	Mozilla 5.0 (Apibot 0.30b5)
		application/vnd.php.serialized
	Freebase Deathbot
		text/..
	EarwigBot/0.1.dev.gitbc9fdf28 (Python/2.7.1; https://github.com/earwig/earwigbot; mail address )
		application/json
		application/x-www-form-urlencoded
	Empedia Bot
		text/..
	wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0
		image/..
		application/json
		text/..
		-
	GNAA-bot
		text/..
	Mozilla/5.0 (Bgbot 0.5)
		text/..
	Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )"
		application/xml
		text/..
	GoogleBot
		text/..
		image/..
	FAST Enterprise Crawler/6.7.8 ( mail address )
		text/..
		-
	Geni ircpybot 1.0
		text/..
		application/json
		application/xml
	infraEnterprise v8 Web Crawler
		-
		text/..
	DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7600.0; )
		text/..
	UniversalFeedParser/5.1.1 https://code.google.com/p/feedparser/
		text/..
		application/xml
		-
	MediaWiki::Bot/v3.4.2
		application/json
	AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
		application/json
	Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
		text/..
		-
	Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.4.152;) Opera Mini/3.1
		image/..
		text/..
		-
	OpenSearchServer_Bot
		text/..
	My Bot
		text/..
		image/..
	Shad robot/1.0
		text/..
	MetallmanulBot for Wiktionary (run by Metallmanul)
		application/json
19527.15	total