Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Jul 2013 - 28 Jul 2013 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

Warning: all recent Wikimedia traffic analysis reports have been generated from old scripts.

The scripts are orphaned, and have not been maintained for at least 6 months. Many bugs are considerably older.
Known Bugzilla bugs: 46190, 46191, 46195, 46201, 46205, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 46289

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 118,889,070 page requests (mime type text/html only!) per day are considered crawler requests, out of 605,766,960 external requests, which is 19.6%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: s~libmuteki2) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9) - -
 code.google.com/p/abottext/..abot v1.0 url - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) de -
 www.google.com/coop/cse/creftext/..PageFetcher-Google-CoOp;((url) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~amort141) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq4) - -
 code.google.com/appenginetext/..Mozilla/5.0 (X11; Linux x86_64; zh_CN) KHTML/4.10.4 Konqueror/4.10 AppEngine-Google; (url; appid: s~42overwall) zh-CN,en-US;q=0.9,en;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5) - -
 code.google.com/appenginetext/..Acre/dev/53:906 factchecker.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kuryproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kauflog) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nmimsforti) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline0) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 KHTML Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (url; appid: s~feedly-social) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 6.1; zh-CN; rv:1.9.2.2) AppEngine-Google; (url; appid: fwall-w15) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kittehserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq0) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline7) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ape-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~triipwiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kickermonkpx) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vi-mobile) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadl6) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appengineimage/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AlexaToolbar/alxg-3.1 AppEngine-Google; (url; appid: s~chauslj) zh-CN,zh;q=0.8 -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy7) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bie99miracle) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp5) - -
 code.google.com/appenginetext/..Acre/dev/59:906 ubiquity.freebaseapps.com AppEngine-Google; (url; appid: s~freebaseapps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d-accord-proxyserver) - -
 code.google.com/appenginetext/..Mozilla/5.0 (X11; Linux x86_64; zh_CN) KHTML/4.10.3 Konqueror/4.10 AppEngine-Google; (url; appid: s~42overwall) zh-CN,en-US;q=0.9,en;q=0.8 -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mrjosiaht) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~italiatiki) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps003) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.4 KHTML Chrome/22.0.1229.94 Safari/537.4 AlexaToolbar/alxg-3.1 AppEngine-Google; (url; appid: s~chauslj) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~scroll-down-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk9-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy4-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk8-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython7-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~bereturning-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf702-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile8-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~cgiproxy5-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: tutorialses) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxy6009) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy5070-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~appcodes-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~response-out-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: bereturning) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6006-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surfproxy4) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmlapplication/jsonGoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-US -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..User-Agent: Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
msn
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) ja-JP,ja;q=0.8,en-US;q=0.6,en;q=0.4 -
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; ) PHP/5.4 - -
 www.digplanet.com/wikitext/..Digplanet/1.0 (url; ) PHP/5.4 - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/text/..WikipediaApp_svc/1.5.13 (url; mail address ) com.orange.mmp/1.7.0 - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
news
 www.news.net/text/..News.Net/1.1 (url; mail address ) BasedOnRawPHP - -
 www.news.net/application/jsonNews.Net/1.1 (url; mail address ) BasedOnRawPHP - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/application/xmlAhrefsBot.Feeds v0.1; url - -
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmapplication/jsonMozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible;Sosospider/2.0;url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmimage/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmapplication/xmlMozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.html-Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_35; Europe/de) url - -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.6.0_41; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-48-generic; java 1.7.0_25; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.5; java 1.6.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_04; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-1.29-default; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; GMT01:00/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_30; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; x86_64 Mac OS X 10.8.4; java 1.6.0_51; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-48-generic; java 1.8.0-ea; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-49-generic; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_02; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-26-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-gentoo-r1; java 1.7.0_21; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_04; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows 8 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-45-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 FreeBSD 9.1-RELEASE-p4; java 1.6.0_32; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.9.9-1-ARCH; java 1.7.0_40; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-49-generic-pae; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.9.7; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-23-generic; java 1.6.0_27; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-358.11.1.el6.x86_64; java 1.6.0_24; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-17-generic; java 1.6.0_27; Europe/sv) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-26-generic; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-26-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-20-pve; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-49-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.9-1-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
pchome
 www.pchome.com.tw/pchomebot.htmtext/..Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
 www.pchome.com.tw/pchomebot.htm-Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
sblog
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test1; url) cs -
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/application/jsonMozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
wordpress
 raymondpronk.wordpress.comtext/..WordPress/3.6-beta4-24612; url - -
 ramanan50.wordpress.comtext/..WordPress/3.6-beta4-24523; url - -
 biblecontroversiesdotcom.wordpress.comtext/..WordPress/3.6-beta4-24534; url - -
 josefboberg.wordpress.comtext/..WordPress/3.6-RC1-24717; url - -
 kittysjones.wordpress.comtext/..WordPress/3.6-RC2; url - -
 josefboberg.wordpress.comtext/..WordPress/3.6-RC2; url - -
 josefboberg.wordpress.comtext/..WordPress/3.6-RC1-24778; url - -
acoon
 www.acoon.de/robot.asptext/..Mozilla/5.0 (compatible; AcoonBot/4.11.1; url) - -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) - -
 shoulu.jike.com/spider.htmlapplication/jsonMozilla/5.0 (compatible; JikeSpider; url) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
wikipedia
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.20.0 url - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
profound
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
crossref
 alm.labs.crossref.orgapplication/jsonArticle Level Metrics - url - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
seoratio
 www.seoratio.de/zyklop-crawler/text/..Mozilla/5.0 (compatible; zyklop; url) - -
wikidict
 www.wikidict.detext/..url - -
webmeup
 webmeup.com/crawler.htmltext/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
sourceforge
 linkchecker.sourceforge.net/text/..LinkChecker/7.4 (url) - -
 linkchecker.sourceforge.net/image/..LinkChecker/7.4 (url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
scrapy
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
 scrapy.orgtext/..Scrapy/0.16.0 (url) en -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ar -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article Level Metrics - url - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
omgili
 omgili.comtext/..omgilibot/0.4 url en-us,en;q=0.5 -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
yunyun
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) - -
 www.yunyun.com/SiteInfo.php?r=abouttext/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/SiteInfo.php?r=aboutimage/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
example
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolTool/application/jsonMyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.com/MyCoolTool/text/..MyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url) - -
 commons.wikimedia.org/wiki/User:Thumbnails_Check_Botimage/..Thumbnails_Check_Bot/0.1 (url; beta) - -
 meta.wikimedia.org/wiki/User:Tietewtext/..Cheebot/0.5.7 (url) - -
 tools.wikimedia.de/~para/GeoCommons/text/..url - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) ja -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
pingdom
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
73
 178.63.73.19application/jsonArticle Level Metrics - url - -
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url) - -
 enwp.org/User:Hellknowz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET) - -
zeerch
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; ZEERCHBOT; url) - -
 www.zeerch.com/bot.phpimage/..Mozilla/5.0 (compatible; ZEERCHBOT; url) - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
holasoyramon
 www.holasoyramon.com/blogtext/..WordPress/3.5.1; url - -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X101.xmlimage/..url en,hi -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
karkka
 karkka.comtext/..url - -
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url) - -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
psu
 citeseerx.ist.psu.edutext/..citeseerxbot (compatible; heritrix/1.14.4 url) - -
 citeseerx.ist.psu.eduimage/..citeseerxbot (compatible; heritrix/1.14.4 url) - -
xbmc
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) - -
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 wummel.github.com/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.4; url) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
automoveisnaweb
 automoveisnaweb.com.br/wordpresstext/..WordPress/3.5.2; url - -
 automoveisnaweb.com.br/wp2text/..WordPress/3.5.2; url - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
weborking
 weborking.comtext/..Weborking(url) - -
 weborking.comapplication/jsonWeborking(url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.com-Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
vermagerd
 www.vermagerd.be/wptext/..WordPress/3.4.2; url - -
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.96.3; url) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
trendiction
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
tkolb
 www.tkolb.de/distsearch.txttext/..DistSearchBot/dev (url) - -
aramabeta
 www.aramabeta.comtext/..betasearch/Nutch-2.1 (Academic Beta Search; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
pride-news
 pride-news.nettext/..WordPress/3.5.1; url - -
 pride-news.nettext/..WordPress/3.5.2; url - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
metamagazine
 metamagazine.comtext/..WordPress/3.5.2; url - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
readability
 readability.com/about/text/..Readability/95e8c2 - url - -
 readability.com/about/text/..Readability/744cab - url - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) - -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
linkdex
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.8 (url) - -
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url) en-us,en;q=0.5 -
tiscali
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.10.2 url) * -
 www.tiscali.it/text/..Mozilla/5.0 (compatible; IstellaBot/1.10.2 url) - -
sf
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url) en-us,en;q=0.5 -
 magpierss.sf.nettext/..MagpieRSS/0.7x (url) en-us,en;q=0.5 -
rockmelt
 rockmelt.comtext/..RockmeltEmbedder (url; mail address ) - -
go
 kc.nict.go.jp/project1/crawl-ja.htmltext/..ICC-Crawler (Mozilla-compatible; mail address ; url) ja -
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
wandex
 wandex.nettext/..Mozilla/5.0 (compatible; World Wide Web Wanderer (Wandex Bot)/1.4; url) - -
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
localhost
 localhost/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - -
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -
hvacrepairmedic
 denver.hvacrepairmedic.comtext/..WordPress/3.5.2; url - -
 aurora.hvacrepairmedic.comtext/..WordPress/3.5.2; url - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) - -
zapbot
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url) - -
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url) - -
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url) - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -
manzia
 www.manzia.com/crawlertext/..ManziaBot/Nutch-2.1 (Web crawler from Manzia Corp; url; mail address ) - -
seomoz
 www.seomoz.org/dp/rogerbottext/..rogerbot/1.0 (url, mail address ) - -
gg
 gg.pl/text/..GG PeekBot 2.0 ( url http://info.gadu-gadu.pl/praca ) - -
 gg.pl/image/..GG PeekBot 2.0 ( url http://info.gadu-gadu.pl/praca ) - -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
142248.620000014total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
Peachy MediaWiki Bot API Version 2.0 (alpha 3) - -
 application/vnd.php.serialized
 text/..
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 text/..
 -
 image/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
wikiwix-bot-3.0 - -
 text/..
 -
LinkParser/2.0 - -
 text/..
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
PythonWikipediaBot/1.0 - -
 application/json
 application/xml
 text/..
 -
GoogleBot-Image/1.0 - -
 text/..
 image/..
 application/ogg
redirect/r11780 Pywikipediabot/2.0 - -
 application/json
redirect/r11761 Pywikipediabot/2.0 - -
 application/json
redirect/r11704 Pywikipediabot/2.0 - -
 application/json
spider - -
 text/..
 application/vnd.php.serialized
 application/json
 image/..
tigerbot - -
 application/json
 text/..
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
 video/webm
redirect/r11743 Pywikipediabot/2.0 - -
 application/json
 text/..
maj_articles_recents/r11700 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
pywikipedia-wikidata_missinglabel_add.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
YisouSpider - -
 text/..
 -
 application/ogg
 application/json
ClueBot/1.1 - -
 application/vnd.php.serialized
Semantix Bot 0.1 - -
 text/..
My Nutch Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
Mozilla/5.0 (compatible; SearchBot) - -
 text/..
ClueBot/2.0 - -
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-interwiki.py/r11655 Pywikipediabot/1.0 - -
 application/xml
 application/json
 text/..
pywikipedia-wikidata_add_label_only.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
www.integromedb.org/Crawler - -
 text/..
 image/..
 -
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 - -
 application/json
redirect/r11779 Pywikipediabot/2.0 - -
 application/json
redirect/r11752 Pywikipediabot/2.0 - -
 application/json
redirect/r11776 Pywikipediabot/2.0 - -
 application/json
redirect/r11725 Pywikipediabot/2.0 - -
 application/json
redirect/r11734 Pywikipediabot/2.0 - -
 application/json
redirect/r11730 Pywikipediabot/2.0 - -
 application/json
redirect/r11716 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
DigitalsmithsBot - -
 text/..
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 image/..
 video/webm
 audio/webm
 application/ogg
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
MediaWiki::Bot/3.2.6 - -
 application/json
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
pywikipedia-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 application/xml
MediaWiki::Bot/5.005006 - -
 application/json
 text/..
p-welcome-d-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
WikiCatResearchBot ( mail address ) - -
 text/..
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
pywikipediabot_old-wikidata_irc_global.py/r11364 Pywikipediabot/1.0 - -
 application/json
Bot: mail address - -
 text/..
pywikipedia-fr.wikt.RemiseEnForme.py/r11692 Pywikipediabot/1.0 - -
 application/json
moje-wymowa.py/r11483 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11762 Pywikipediabot/1.0 - -
 application/json
Tawbot (public svn release; plwiki) - -
 text/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
botjagwar-crawlrandom.py/r11215 Pywikipediabot/1.0 - -
 application/json
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
WikidataBot framework - -
 application/json
 text/..
new2-zzdi1.py/r11452 Pywikipediabot/1.0 - -
 application/json
 application/xml
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
fa-welcome.py/r11054 Pywikipediabot/1.0 - -
 application/json
pywikipedia-radeh.py/r11655 Pywikipediabot/1.0 - -
 application/json
 application/xml
Java bot - -
 application/json
 text/..
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
 -
pywikipedia3-MakeScoreTable_test2.py/r11448 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 text/..
pywikipedia-featured.py/r11674 Pywikipediabot/1.0 - -
 application/json
 application/xml
GermCrawler - -
 application/json
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider; zh-CN -
 text/..
 -
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 -
pywikipedia-redirect.py/r11760 Pywikipediabot/1.0 - -
 application/json
Test Webbot - -
 text/..
pywikipedia-wikidata_setproperty.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
moje-porzucone.py/r11483 Pywikipediabot/1.0 - -
 application/json
 -
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
pywikipedia-redirect.py/r11731 Pywikipediabot/1.0 - -
 application/json
pywikipedia-dave_sezioni.py/r11522 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11735 Pywikipediabot/1.0 - -
 application/json
pyreza-zzfatranslator.py/r11749 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
DotNetWikiBot/2.100 (Unix 3.2.0.48; ) - -
 text/..
milog_bot/1.0 ( mail address ) - -
 text/..
pywikipedia2-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
 text/..
importer/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
wikidata-sitelinks.py/r11618 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
My Nutch Spider/Nutch-2.0 - -
 text/..
 image/..
 application/ogg
WikiBot/0.1 - -
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
COIBot/1.00 - -
 text/..
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
Sogou web spider/4.0 zh-cn -
 text/..
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
pywikipedia-featured.py/r11747 Pywikipediabot/1.0 - -
 application/json
 application/xml
DeletionBot-deletion.py/r11525 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-WPP.py/r11746 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11745 Pywikipediabot/1.0 - -
 application/json
Webbull-Bot - -
 text/..
bot: fr-anal - -
 application/json
Desktop-test.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia-WPP.py/r11618 Pywikipediabot/1.0 - -
 application/json
 application/xml
COIBot/2.0 - -
 text/..
botjagwar-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
pywikipediabot_old-wikidata_new_global.py/r11364 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-dobrodoslice.py/r11692 Pywikipediabot/1.0 - -
 application/json
Epywikipedia-welcome.py/r11181 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
 application/xml
pywikipedia-28main.build89.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikirish-welcome.py/r11215 Pywikipediabot/1.0 - -
 application/json
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
360spider-image - -
 image/..
 text/..
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
Empedia Bot - -
 text/..
burst bot - -
 text/..
Mozilla/5.0 (compatible; MyBot/1.0;) - -
 application/json
pywikipedia-zzinter.py/r11655 Pywikipediabot/1.0 - -
 application/json
 application/xml
aleph-avbot.py/r11368 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-redirect.py/r11763 Pywikipediabot/1.0 - -
 application/json
request-daemon/r11515 Pywikipediabot/2.0 - -
 application/json
ko_missing/r11703 Pywikipediabot/2.0 - -
 application/json
www.monit24.pl-m24Bot/4.1- - -
 image/..
 text/..
wdaka/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11751 Pywikipediabot/1.0 - -
 application/json
 text/..
HTMLParser/2.0 - -
 text/..
 application/json
pywikipedia-redirect.py/r11711 Pywikipediabot/1.0 - -
 application/json
UAA-UAA.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11681 Pywikipediabot/1.0 - -
 application/json
 application/xml
SurakWare MediaWiki Bot/1.0 - -
 text/..
 application/xml
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
BotReversor-BotReversor.py/r11604 Pywikipediabot/1.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
FAST Enterprise Crawler 6 used by MS ( mail address ) - -
 text/..
pywikipedia-importwdwi.py/r11746 Pywikipediabot/1.0 - -
 application/json
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
pywikipedia-featured.py/r11700 Pywikipediabot/1.0 - -
 application/json
 application/xml
botjagwar-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
JavaCrawler/1.1 - -
 text/..
Addbot Wikimedia Bot - -
 text/..
 application/vnd.php.serialized
 -
pywikipedia-commonscat.py/r11692 Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-VM-auto-erl.py/r11602 Pywikipediabot/1.0 - -
 application/json
pywikipedia-importwdwi.py/r11597 Pywikipediabot/1.0 - -
 application/json
pywikipedia-radeh.py/r11749 Pywikipediabot/1.0 - -
 application/json
 application/xml
Wikipediabot-wikidata_begriffsklaerung.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia2-redirect.py/r11762 Pywikipediabot/1.0 - -
 application/json
Projet IW-PIW2.py/r10533 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-welcome-custom.py/r11354 Pywikipediabot/1.0 - -
 application/json
Simple Page Crawler. Contact: mail address en-us -
 text/..
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
pywikipedia-wikidata_coordinaten_dms2.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r11700 Pywikipediabot/2.0 - -
 application/json
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
FTRF: Friendly robot/1.3 - -
 text/..
pywikipedia-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
XLinkBot/1.00 - -
 text/..
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikipedia-basicarticledisamblist.py/r11681 Pywikipediabot/1.0 - -
 application/json
2418791/r11657 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
pywikipedia-redirect.py/r11692 Pywikipediabot/1.0 - -
 application/json
 text/..
Wiki.java 0.27 r137 (OctraBot 2.8) - -
 text/..
 application/json
pyiw-redirect.py/r11641 Pywikipediabot/1.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129b Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
Python27-pythonw.exe/r10533 Pywikipediabot/1.0 - -
 application/json
 application/xml
2316607/r11657 Pywikipediabot/2.0 - -
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
https://www.wikidata.org/wiki/User:Legobot - -
 text/..
logpatrol/r11761 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11707 Pywikipediabot/1.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129c Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
unknown, 2a00:4b00:13d:200::129f Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
User-Agent Baiduspider - -
 text/..
 image/..
 -
logpatrol/r11725 Pywikipediabot/2.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129a Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
unknown, 2a00:4b00:13d:200::129e Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
BotReversor-BotReversor.py/r11707 Pywikipediabot/1.0 - -
 application/json
pywikipedia-importwd.py/r11597 Pywikipediabot/1.0 - -
 application/json
ReaperBot/2.0.0 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot> - -
 text/..
pywikipedia-zzinter.py/r11749 Pywikipediabot/1.0 - -
 application/json
pywikipedia-com_cat.py/r11692 Pywikipediabot/1.0 - -
 application/json
pywikipedia2-redirect.py/r11760 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11707 Pywikipediabot/1.0 - -
 application/json
Mozilla 5.0 (Apibot 0.32) - -
 application/vnd.php.serialized
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 text/..
FAST Search Web Crawler 14.0.0325.0000 - -
 text/..
 -
 application/ogg
py-commonscat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-com_cat.py/r11760 Pywikipediabot/1.0 - -
 application/json
harvest_template/r11694 Pywikipediabot/2.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129d Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
pywikipedia-com_cat.py/r11775 Pywikipediabot/1.0 - -
 application/json
theWxitBot/0.1 - -
 application/json
 image/..
Kompakt-bot (by user:kompakt) - -
 application/json
datetemplates/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11714 Pywikipediabot/1.0 - -
 application/json
Zing-BottaBot/2.0 - -
 text/..
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9 en-us,en;q=0.5 -
 image/..
 text/..
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
 text/..
admin-ipblocker-auto.py/r11723 Pywikipediabot/1.0 - -
 application/json
pywikipedia-MakeScoreTable.py/r11747 Pywikipediabot/1.0 - -
 application/json
iw2data/r11703 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en,* -
 image/..
 application/json
 text/..
pywikipedia-redirect.py/r11385 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikilab-llista_mils_2013.py/r11707 Pywikipediabot/1.0 - -
 application/json
pywikipedia-archivebot.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-17-progreso1911.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
pywikipedia-featured.py/r11675 Pywikipediabot/1.0 - -
 application/json
Wikipediabot-wikidata_gndcheck.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
 -
rg/r11515 Pywikipediabot/2.0 - -
 application/json
Global-Crawler/Nutch-1.4 (Search platform crawler) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
harvest_template/r11712 Pywikipediabot/2.0 - -
 application/json
dtSearchSpider - -
 text/..
Python27-pythonw.exe/r11618 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 bg -
 image/..
 text/..
pywikipedia-com_cat.py/r11723 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11762 Pywikipediabot/1.0 - -
 application/json
 application/xml
newser-wikinewser2.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11731 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
featured/r11734 Pywikipediabot/2.0 - -
 application/json
artists/r11515 Pywikipediabot/2.0 - -
 application/json
AdMedia bot - -
 text/..
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
Wiki.java 0.27 r137 (OctraBot 2.7) - -
 text/..
 application/json
drp_warn/r11683 Pywikipediabot/2.0 - -
 application/json
 text/..
wd_import/r11653 Pywikipediabot/2.0 - -
 application/json
pywikipedia-redirect.py/r11723 Pywikipediabot/1.0 - -
 application/json
Pywikipedia-new_interwiki.py/r11694 Pywikipediabot/1.0 - -
 application/json
 application/xml
matilda/r11686 Pywikipediabot/2.0 - -
 application/json
update-task-categories/r11515 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
pywikipedia-redirect.py/r11746 Pywikipediabot/1.0 - -
 application/json
VWBot - -
 application/json
GoogleBot - -
 text/..
wikiscore-MakeScoreTable.py/r11549 Pywikipediabot/1.0 - -
 application/json
WPBot 1.0 - -
 text/..
python-wikitools/1.2 (User:LaraBot) - -
 application/json
pywikipedia-wd_move.py/r11760 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_coordinaten_dms.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
viaf-sbn_reconciler/r11276 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
pywikipedia-redirect.py/r11710 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 -
pywikipedia-pwb.py/r11762 Pywikipediabot/1.0 - -
 application/json
 image/..
 application/xml
 text/..
EarwigBot/0.2.dev.git5931f375 (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
pywikipedia-featured.py/r11706 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wd_move.py/r11723 Pywikipediabot/1.0 - -
 application/json
hercule-commonscat.py/r11762 Pywikipediabot/1.0 - -
 application/json
touch/r11635 Pywikipediabot/2.0 - -
 application/json
pywikipedia2-redirect.py/r11735 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11760 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.102 (Unix 3.2.0.48; ) - -
 text/..
hercule-commonscat.py/r11735 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikipedia2-redirect.py/r11707 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r11735 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11713 Pywikipediabot/1.0 - -
 application/json
hercule-commonscat.py/r11731 Pywikipediabot/1.0 - -
 application/json
EPYWIK~1-welcome.py/r11181 Pywikipediabot/1.0 - -
 application/json
pywikipedia-redirect.py/r11712 Pywikipediabot/1.0 - -
 application/json
hercule-commonscat.py/r11711 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-wikidata_disamb_dp_labelonly.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-featured.py/r11745 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-wikidata_coordinaten_bouwwerken.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
HopperBot - -
 image/..
 text/..
pywikipedia-basicdisambrefslist.py/r11681 Pywikipediabot/1.0 - -
 application/json
 application/xml
('python-wikitools/1.1.1 (User:BernsteinBot)',) - -
 application/json
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
pywikipedia-redirect.py/r11706 Pywikipediabot/1.0 - -
 application/json
Cyberfox Spider ru, en -
 text/..
MediaWiki Snake Robot/1.1 - -
 application/json
Projet Sport-Projetsportliste2.py/r10533 Pywikipediabot/1.0 - -
 application/json
pywikipedia2-redirect.py/r11745 Pywikipediabot/1.0 - -
 application/json
pywikipedia-nedavne.py/r11692 Pywikipediabot/1.0 - -
 application/json
 text/..
bitlybot - -
 text/..
 image/..
MediaWiki::Bot 3.1.5 - -
 application/json
opentask-opentasks.py/r10326 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.102 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
pywikipedia-wikidata_setproperty_frominfobox_nrhp.py/r11200 Pywikipediabot/1.0 - -
 application/json
trunk-featured.py/r11714 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-Leer.py/r11780 Pywikipediabot/1.0 - -
 application/json
hercule-commonscat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-redirect.py/r11708 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
HTMLParser/1.6 - -
 text/..
pywikipedia2-redirect.py/r11714 Pywikipediabot/1.0 - -
 application/json
Trad_maintenance/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
SkimBot/1.0 (www.skimlinks.com mail address ) en-us,en;q=0.5 -
 text/..
Robot911 - -
 text/..
 image/..
YBot/0.1 - -
 application/vnd.php.serialized
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
ZeBigWebBot - -
 text/..
pywikipedia-fixing_redirects (4).py/r11650 Pywikipediabot/1.0 - -
 application/json
 application/xml
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 -
 text/..
 image/..
mrajedrez-category_redirect.py/r11321 Pywikipediabot/1.0 - -
 application/json
35579.35total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, Oct 4, 2013 12:45
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers