Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Feb 2014 - 28 Feb 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 121,507,820 page requests (mime type text/html only!) per day are considered crawler requests, out of 611,442,430 external requests, which is 19.9%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmtext/..37.58.85.180 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htm-(compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)Accept: text/html,application/xhtmlxml,application/xml;image/png,image/jpeg,image/*;q=0.9,*/*;q=0.8 en-us;q=0.7, en;q=0.3 -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) - -
 developers.facebook.com-facebookplatform/1.0 (url) -
 developers.facebook.com-facebookplatform/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~certificate-check-403) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~thingswebdev) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: s~certificate-check-403) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~dizykit) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ageryder) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 KHTML Chrome/32.0.1700.76 Safari/537.36 AppEngine-Google; (url; appid: s~zhuluchi) zh-CN,zh;q=0.8,zh-TW;q=0.6 -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 KHTML Chrome/32.0.1700.76 Safari/537.36 AppEngine-Google; (url; appid: s~zhuluchi) zh-CN,zh;q=0.8,zh-TW;q=0.6 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: easypox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tinyproxys) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icebre4ker) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: javahavens) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kakuro69) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~harunakaze) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vadim-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 KHTML Chrome/32.0.1700.102 Safari/537.36 AppEngine-Google; (url; appid: s~nsftcn) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 KHTML Chrome/32.0.1700.102 Safari/537.36 AppEngine-Google; (url; appid: s~nsftcn) zh-CN,zh;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lukaviy419) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icecool1987) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: finchproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~tinyproxys) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~warawasetekure) - -
 code.google.com/appengine-Mozilla/5.0 (Windows NT 6.1; rv:27.0) Gecko/20100101 Firefox/27.0 AppEngine-Google; (url; appid: s~qcyao198701) zh-cn,en-us;q=0.5 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: randohnson) - -
 www.google.com/bot.htmlapplication/pdfGoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..37.58.85.180 Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.html-(compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/opensearchdescription+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlapplication/rsd+xmlMozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmlimage/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..37.58.85.180 msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-(compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..37.58.85.180 Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.el6.x86_64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.7.10-1.24-desktop; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-35-generic; java 1.7.0_51; Asia/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_51; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.7.10-1.24-desktop; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-12-generic; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-45-generic; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-17-generic; java 1.7.0_51; Asia/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_51; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.7; java 1.7.0_51; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_40; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.5-1-ARCH; java 1.7.0_45; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12-1-amd64; java 1.7.0_51; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.9-2-ARCH; java 1.7.0_51; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-35-generic; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_51; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows Server 2008 R2 6.1; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-45-generic; java 1.7.0_51; Asia/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (allip/any; i386 Linux 2.6.32-26-pve; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.6-1-CHAKRA; java 1.7.0_45; Canada/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.10-300.fc20.x86_64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_40; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-34-generic; java 1.6.0_27; Africa/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-4-686-pae; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.13.0-2-ARCH; java 1.7.0_51; GMT/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-12-generic; java 1.7.0_51; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmlimage/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.6.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.6-gentoo; java 1.6.0_27; Australia/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Asia/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.3.1.el6.x86_64; java 1.6.0_30; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo20130930; java 1.7.0_45; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_51; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-58-generic; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-15-generic; java 1.6.0_27; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.5.1.el6.x86_64; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_51; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; GMT-05:00/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 3.2.0-54-generic; java 1.6.0_27; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-45-generic; java 1.6.0_27; Europe/uk) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-56-generic; java 1.6.0_27; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-36-generic; java 1.7.0_51; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (allip-any; amd64 Linux 3.8.0-35-generic; java 1.7.0_51; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmlimage/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.0-15-generic; java 1.7.0_51; Asia/en) url en-us,en;q=0.5 -
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.8.7; java 1.7.0_51; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_21; America/pt) url en-us,en;q=0.5 -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/robot/text/..37.58.85.180 Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/application/xmlAhrefsBot.Feeds v0.1; url - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07image/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou Pic Spider/3.0(url) zh-cn -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url) - -
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) - -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test1; url) cs -
 fulltext.sblog.cz/-Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
wikipedia
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentation-WPCleaner (url) - -
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.22.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggle-Huggle/2.1.22.0 url - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggle-Huggle/2.1.21.0 url - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBot-OrlodrimBot/1.0 (url) -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.22.0 url -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBotapplication/x-www-form-urlencodedOrlodrimBot/1.0 (url) -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
wordpress
 menschlein.wordpress.comtext/..WordPress/3.8.1; url - -
 y35pm.wordpress.comtext/..WordPress/3.8.1; url - -
 tsjok45.wordpress.comtext/..WordPress/3.8.1; url - -
 wildanrenaldi.wordpress.comtext/..WordPress/3.8.1; url - -
 whatownsme.wordpress.comtext/..WordPress/3.8.1; url - -
 02varvara.wordpress.comtext/..WordPress/3.8.1; url - -
 pasadenaartbeat.wordpress.comtext/..WordPress/3.8.1; url - -
 raymondpronk.wordpress.comtext/..WordPress/3.8.1; url - -
 halokue.wordpress.comtext/..WordPress/3.8.1; url - -
 williamstickevers.wordpress.comtext/..WordPress/3.8.1; url - -
 manofroma.wordpress.comtext/..WordPress/3.8.1; url - -
 dearkitty1.wordpress.comtext/..WordPress/3.8.1; url - -
 homepaddock.wordpress.comtext/..WordPress/3.8.1; url - -
 pronkpops.wordpress.comtext/..WordPress/3.8.1; url - -
 hrexach.wordpress.comtext/..WordPress/3.8.1; url - -
 barbielistholland.wordpress.comtext/..WordPress/3.8.1; url - -
 cuthulan.wordpress.comtext/..WordPress/3.8.1; url - -
 melusinefee.wordpress.comtext/..WordPress/3.8.1; url - -
 drstvmovies.wordpress.comtext/..WordPress/3.8.1; url - -
 ramanan50.wordpress.comtext/..WordPress/3.8.1; url - -
 thesandymonocle.wordpress.comtext/..WordPress/3.8.1; url - -
 gunnyg.wordpress.comtext/..WordPress/3.8.1; url - -
 anarchyisorder.wordpress.comtext/..WordPress/3.8.1; url - -
 omadeon.wordpress.comtext/..WordPress/3.8.1; url - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) ru -
 www.majestic12.co.uk/bot.php?text/..(compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?application/jsonMozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
 nodejs.orgapplication/jsonNodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) fr,en-us,en;q=0.5 -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) - -
crossref
 alm.labs.crossref.orgapplication/jsonArticle Level Metrics - url -
bin-co
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url -
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-57-g180fc98 (url) en -
 scrapy.org-Scrapy/0.23.0-57-g180fc98 (url) en -
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.22.1 (url) en -
 scrapy.org-Scrapy/0.22.1 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-77-gc99f1d2 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
 scrapy.orgtext/..Scrapy/0.22.2 (url) en -
 scrapy.org-Scrapy/0.23.0-77-gc99f1d2 (url) en -
 scrapy.orgtext/..Scrapy/0.22.0 (url) de -
 scrapy.orgtext/..Scrapy/0.18.4 (url) en -
localhost
 localhost/clumpit/bot.phptext/..Mozilla/5.0 (compatible; clumboot; url) - -
 localhost/wordpresstext/..WordPress/3.8.1; url - -
seoratio
 www.seoratio.de/zyklop-crawler/text/..Mozilla/5.0 (compatible; zyklop; url) - -
 www.seoratio.de/zyklop-crawler/-Mozilla/5.0 (compatible; zyklop; url) - -
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.97; url) - -
flipboard
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
 www.phonifier.com-Mozilla/5.0 (compatible; Phonifier; url) - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/-Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address es-es -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ru -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics - url - -
webpronews
 www.webpronews.comtext/..WordPress/3.8.1; url - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
diffbot
 www.diffbot.com-Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
 www.diffbot.comimage/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
 www.diffbot.comtext/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
mapsway
 mapsway.com/text/..MapswayBot/1.0 (url) - -
example
 example.comapplication/jsonArticle-Level Metrics - url - -
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolTool/text/..User-Agent: MyCoolTool/1.1 (url; mail address ) - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.com/MyCoolTool/application/jsonMyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
netarkivet
 netarkivet.dk/webcrawler/text/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/image/..Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
 netarkivet.dk/webcrawler/-Mozilla/5.0 (compatible; heritrix/1.14.4 url) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
 pinterest.com/-Pinterest/0.1 url - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.comtext/..FeedlyApp/1.0 (url) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
gov
 pandora.nla.gov.au/crawl.htmltext/..Mozilla/5.0 (compatible; archive.org_bot/3.3.0 url) - -
 pandora.nla.gov.au/crawl.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot/3.3.0 url) - -
 pandora.nla.gov.au/crawl.html-Mozilla/5.0 (compatible; archive.org_bot/3.3.0 url) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - -
archive
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bot-Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
xbmc
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.2;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Windows NT 6.1; url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (iOS; 13.0.0, Version 6.1.4 (Build 10B809); url) - -
 www.xbmc.orgimage/..XBMC/12.3 Git:20131212-9ed3e58 (Linux; Android; 3.0.36 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130527-a36a92b (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1;WOW64;Win64;x64; url) - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) -
 toolserver.org/~dispenser/text/..DispensersTools (url) -
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) ja, en -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
vote-often
 vote-often.comtext/..WordPress/3.5.1; url - -
http:
 http://www.goguide-directory.com.au/application/vnd.php.serializedTesting for url - -
 http://alm.example.orgapplication/jsonArticle Level Metrics - url - -
 http://www.goguide-directory.com.au/application/vnd.php.serializedurl - -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
site
 www.site.com/application/jsonBot/2.0 (url) -
wesee
 www.wesee.com/en/support/bot/image/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
 www.wesee.com/en/support/bot/text/..WeSEE:Search/0.1 (Alpha, url) - -
automoveisnaweb
 automoveisnaweb.com.br/uk3text/..WordPress/3.8.1; url - -
 automoveisnaweb.com.br/uk4text/..WordPress/3.8; url - -
 automoveisnaweb.com.br/uk6text/..WordPress/3.8.1; url - -
 automoveisnaweb.com.br/uk5text/..WordPress/3.8.1; url - -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo-crawler; url; mail address ) - -
 www.parsijoo.ir/image/..Mozilla/5.0 (compatible; parsijoo-crawler; url; mail address ) - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
 www.rcdtokyo.com/pc2m/-Mozilla/5.0 (compatible; PEAR HTTP_Request class; url) ja -
8
 192.168.0.8/bot.phptext/..Mozilla/5.0 (compatible; madnatter; url) - -
 192.168.0.8/bot.php-Mozilla/5.0 (compatible; madnatter; url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
wikidict
 www.wikidict.detext/..url - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
github
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
 wummel.github.io/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.6; url) - -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
axtaris
 axtaris.orgtext/..Axtaris Web Crawler/Axtaris-1.0.1 (Hello from Axtaris.org | Web-Crawler; url; mail address ) - -
 axtaris.orgtext/..Axtaris Web Crawler/Axtaris-1.0.1 (Hello from Axtaris.org | Web-Crawler; url; mail address ) - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
tu-darmstadt
 www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/text/..Mozilla/5.0 (compatible; kdslbot/3.1.1.4 url) - -
 www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/-Mozilla/5.0 (compatible; kdslbot/3.1.1.4 url) - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) ja -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
 www.grapeshot.co.uk/crawler.phpimage/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
bibalex
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
trendiction
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
 www.proximic.com/info/spider.phpimage/..Mozilla/5.0 (compatible; proximic; url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
 support.paper.li/entries/20023257-what-is-paper-li-Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
Anonymouse
 Anonymouse.org/text/..url (Unix) - -
 Anonymouse.org/image/..url (Unix) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
jyvee
 www.jyvee.comapplication/xmlFeedFetcher; url; v.1.3.79; - -
 www.jyvee.comtext/..FeedFetcher; url; v.1.3.79; - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
qbqm
 www.qbqm.fr/text/..QBQM/1.0 (url) fr -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url) -
 meta.wikimedia.org/wiki/User:Tietewtext/..Cheebot/0.5.7 (url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
 www.netvibes.comapplication/jsonNetvibes (url) de-de -
rebelmouse
 rebelmouse.comimage/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.comtext/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.com-RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
nvdev
 qa.nvdev.comtext/..Netvibes (url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
superfeedr
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.com-Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
easybib
 content.easybib.com/autocite/text/..EasyBib AutoCite (url) - -
 content.easybib.com/autocite/application/jsonEasyBib AutoCite (url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
domain
 yourwebsite.domain.com/text/..Mozilla/5.0 (compatible; heritrix/1.8.0 url) - -
 yourwebsite.domain.com/-Mozilla/5.0 (compatible; heritrix/1.8.0 url) - -
 yourwebsite.domain.com/image/..Mozilla/5.0 (compatible; heritrix/1.8.0 url) - -
uni-hannover
 crawler.dcsec.uni-hannover.detext/..Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 KHTML Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: url - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
dbpedia
 dbpedia.orgtext/..139.18.2.38 DBpedia Sync - url - mail address -
 dbpedia.orgtext/..DBpedia Sync - url - mail address -
linkdex
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
netnewswireapp
 netnewswireapp.com/mac/application/xmlNetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
gorgor
 gorgor.irtext/..hivaBot/hivaBot-0.2 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 gorgor.irtext/..hivaBot(url and mail address )/hivaBot-0.2 en-us,en-gb,en;q=0.7,*;q=0.3 -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.0.7; url ) - -
watchmouse
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
thinglink
 www.thinglink.com/application/jsonMozilla/5.0 (compatible; Thinglink/1.0; url, mail address ) - -
mystudent
gavinr
 gavinr.com/mapwikipedia/application/jsonMapWikipedia/1.1 (url; mail address ) - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.11 (url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) it -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
QtWeb
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
queryseeker
 queryseeker.com/bot.htmltext/..QuerySeekerSpider ( url ) - -
com?type=714647&fr=spigot-yhp-ch
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
backgroundswitcher
 www.backgroundswitcher.com/image/..John's Background Switcher 4.8 (url) - -
 www.backgroundswitcher.com/text/..John's Background Switcher 4.8 (url) - -
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url) fr, en; q=0.3 -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
nowsfeed
 feeding.nowsfeed.comtext/..WordPress/3.8.1; url - -
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) - -
 ireplay.fr/text/..iReplay/1.0 (url) -
aonju
bit
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) en
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) de
vt
 www.cs.vt.eduimage/..Mozilla/5.0 (compatible; heritrix/3.1.0 url) - -
 www.cs.vt.edutext/..Mozilla/5.0 (compatible; heritrix/3.1.0 url) - -
moz
 moz.com/help/pro/what-is-rogerbot-text/..rogerbot/1.0 (url, mail address ) - -
yunyun
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) zh-cn;q=0.8, *;q=0.5 -
image2play
 image2play.com/image/..Mozilla/5.0 (compatible; Image2play/0.1; url) - -
ac
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
 webarchiv.onb.ac.atimage/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
servicesonrequest
 www.servicesonrequest.comtext/..User-Agent: WikiBot (url) - -
 www.servicesonrequest.com/application/jsonWikiBot(url) - -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.3 (url) -
tivine
 tivine.com/application/jsonTiVipedia (url; mail address ) Tivipedia App -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url) en-us -
datoid
 datoid.cz/tag-information/application/jsonDatoid/0.9 (url; mail address ) BasedOnNette/2.0.9 - -
165932.830000013total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 7) - -
 application/vnd.php.serialized
 text/..
 -
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
 application/json
LinkParser/2.0 - -
 text/..
 application/pdf
Peachy MediaWiki Bot API Version 2.0 (alpha 6) - -
 application/vnd.php.serialized
 text/..
 -
tigerbot - -
 -
 application/json
 text/..
 image/..
 application/ogg
maj_articles_recents/rg2754 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
www.integromedb.org/Crawler - -
 text/..
 application/xml
 image/..
 application/json
 application/ogg
wikiwix-bot-3.0 - -
 text/..
pywikipedia-rad1.py/r10694 Pywikipediabot/1.0 - -
 application/json
 text/..
 -
 application/xml
pywikipedia-rad1.py/r10694 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
ClueBot/1.1 - -
 application/vnd.php.serialized
maj_articles_recents/r2607 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
harvest_nltaxo/r2458 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
redirect/rg2764 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
GoogleBot-Image/1.0 - -
 text/..
 image/..
 application/json
 -
pwb/rg2746 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/r2669 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 text/..
 -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
PythonWikipediaBot/1.0 - -
 application/json
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
 image/..
 application/ogg
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
maj_articles_recents/rg2750 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-1newar25.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 -
 text/..
 application/x-www-form-urlencoded
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 -
 application/ogg
 image/..
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
 -
redirect/rg2770 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
ClueBot/2.0 - -
 application/vnd.php.serialized
 text/..
pywikibot-compat-commonscat.py/r10751 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/rg2772 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
py-radehh.py/r11763 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
 -
pywikipedia-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywikibot-1newar25.py/r10860 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikipedia-fr.wikt.RemiseEnForme.py/r68 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 text/..
 application/xml
pywikibot-1newar25.py/r10858 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
redirect/rg2753 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
MediaWiki::Bot/5.005006 - -
 application/json
 -
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
Semantix Bot 0.1 - -
 text/..
MediaWiki::Bot/3.2.6 - -
 application/json
 text/..
redirect/rg2754 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
DigitalsmithsBot - -
 text/..
BOT/0.1 (BOT for JCE) - -
 text/..
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
Twitterbot/1.0 - -
 text/..
 image/..
 -
 application/pdf
 application/ogg
 application/json
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 -
 application/json
 text/..
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; SputnikBot/2.2) ru,en;q=0.1,*;q=0.01 -
 text/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
MediaWiki::Bot/3.005002 - -
 application/json
 text/..
harvest_nltaxo/rg2726 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikibot-1newar25.py/r10854 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
spider - -
 text/..
 application/json
 application/vnd.php.serialized
 image/..
 -
pywikibot-1newar25.py/r10836 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
JackBot-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
PythonWikipediaBot/1.0 -
 application/json
 -
 text/..
DotNetWikiBot/3.0 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/x-www-form-urlencoded
redirect/rg2726 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
 application/json
redirect/rg2765 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
 application/x-www-form-urlencoded
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.54; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
pwb/r2665 Pywikipediabot/2.0 - -
 application/json
redirect/rg2761 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
pywikibot-1newar25.py/r10856 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pwb/rs3792 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
redirect/rg2763 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-checkimages.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
ContextAd Bot 1.0 - -
 text/..
 image/..
redirect/rg2725 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/3.10 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/x-www-form-urlencoded
360spider-image - -
 image/..
 text/..
compat-redirect.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
redirect/rg2768 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/rg2720 Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
anagrama/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
CaBot - -
 text/..
 -
 application/ogg
 unknown/unknown
milog_bot/1.0 ( mail address ) - -
 text/..
pwb/rs3781 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
 text/..
pywikibot-1newar25.py/r10841 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 application/rsd+xml
compat-featured.py/r10734 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
WikidataBot framework - -
 application/json
gsa-crawler (Enterprise; T4-KAWM9JCULZJS9; mail address , mail address ) - -
 text/..
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
 application/x-www-form-urlencoded
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
Peachy MediaWiki Bot API - -
 application/vnd.php.serialized
 text/..
 -
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 - -
 application/json
 -
 text/..
pywikipedia-redirect.py/r10311 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Webbot - -
 text/..
pwb/rs3754 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-radeh.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
compat-featured.py/r10441 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
pywiki-cfd.py/r10360 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
wikiparser/1 CFNetwork/673.0.3 Darwin/13.0.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
pywikibot-1newar25.py/r10853 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider; zh-CN -
 text/..
 -
createHierarchy/rg2726 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.100 (Unix 3.2.0.56; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
 text/..
internationalization/r2631 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-zzredirectyeh.py/r10582 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
pywikibot-1newar25.py/r10828 Pywikipediabot/1.0 - -
 application/json
 -
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
 text/..
SentiBot mail address (compatible with GoogleBot) - -
 text/..
 application/xml
redirect/rg2746 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
porzucone/rg2764 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rs3798 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pwb/r2471 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 text/..
pywikibot-compat-commonscat.py/r10843 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
 application/x-www-form-urlencoded
pywikipediabot-replace.py/r10589 Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 application/xml
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
GermCrawler - -
 application/json
 text/..
 application/xml
redirect/rg2738 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-pwb.py/r10828 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikipedia-git-maghalesaz.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Yeobot/1.1 Contact: mail address ) - -
 text/..
pywikibot-compat-aaa_elenco.py/r10761 Pywikipediabot/1.0 - -
 application/json
 -
strucr.com crawler 2.21.360 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
redirect/rg2750 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
 -
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
 -
pwb/rs3782 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rg2764 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rg2754 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/x-www-form-urlencoded
 application/json
chrwikt/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-wikifyer.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
redirect/rg2756 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
redirect/rg2752 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-feedWikidata-en.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
WorldOfMusicBetaBot/1.0 - -
 text/..
redirect/rg2751 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rg2761 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 -
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
 -
pywikibot-1newar25.py/r10848 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
mail address mail address \xE2\x80\x93 MediaWiki Tcl Bot Framework 0.5 - -
 application/x-www-form-urlencoded
Spinuf Spider - -
 text/..
Wikibot/2.1.1 CFNetwork/672.0.8 Darwin/14.0.0 en-us -
 image/..
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-1newar25.py/r10829 Pywikipediabot/1.0 - -
 application/json
 -
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 -
 text/..
 application/x-www-form-urlencoded
pywikipedia-git-wdph94.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-com_wrong_info.py/r10809 Pywikipediabot/1.0 - -
 -
 application/json
 text/..
mail address - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzinter.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
wiktionary-z_paginasrequeridas.py/r10529 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
redirect/rg2714 Pywikipediabot/2.0 - -
 application/json
pwb/r2526 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
 -
 text/..
createHierarchy/rg2764 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
DotNetWikiBot/2.101 (Unix 3.2.0.54; ) - -
 text/..
 application/xml
 application/x-www-form-urlencoded
DotNetWikiBot/2.100 (Unix 3.2.0.58; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
pwb/rs3776 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
COIBot/1.00 - -
 text/..
 -
pywikibot-compat-aaa_categorieAutori.py/r10761 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
ko_missing/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
replace/r11276 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
HTMLParser/2.0 - -
 text/..
 -
 image/..
pywiki-newar25.py/r10825 Pywikipediabot/1.0 - -
 application/json
 -
make_category_data_snapshot/r2160 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-basictemplatedisamblist.py/r10843 Pywikipediabot/1.0 - -
 application/json
 text/..
 -
compat-basictemplatedisamblist.py/r10860 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; BeetleBot; ) - -
 text/..
 image/..
 application/ogg
 application/json
compat-basictemplatedisamblist.py/r10860 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
compat-basictemplatedisamblist.py/r10843 Pywikipediabot/1.0 -
 -
mijnbots-OnvolDP.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pwb/rg2770 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
strucr.com crawler 2.22.361 (refer to in robots.txt as strucr, see https://strucr.com/bot) - -
 text/..
 -
make_category_data/r2160 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
addDescGemeinde/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-zzporb-main.py/r10694 Pywikipediabot/1.0 -
 -
redirect/rg2767 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
 application/xml
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
 application/xml
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - -
 application/json
 -
pywikipedia-zzporb-main.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 text/..
pwb/rg2753 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,*
 image/..
 text/..
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 - -
 application/json
 application/xml
 -
 text/..
pwb/rs3748 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg2772 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
2471197/rg2726 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
cwikt-interwiki.py/r10825 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rs3753 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2764 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rg2725 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/rg2757 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
porzucone/rg2726 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
porzucone/rg2754 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg2770 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-feedWikidataWithRDFFile.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
 text/..
2649956/rg2764 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-17-progreso1911.py/r10311 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pwb/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rs3779 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
broken-link-crawler - -
 text/..
pwb/rs3789 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
DotNetWikiBot/3.0 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/x-www-form-urlencoded
py-replace.py/r10589 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
porzucone/rg2753 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pyreza-trans.py/r10778 Pywikipediabot/1.0 - -
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
addDescCat/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-git-importwdwiswv.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
pywikipedia-zz_make_cat.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
pwb/rg2752 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywiki-mokhtar.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - -
 application/xml
 text/..
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
 text/..
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
aleph-avbot.py/r11368 Pywikipediabot/1.0 -
 application/json
 text/..
 -
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
pyreza-trans.py/r10778 Pywikipediabot/1.0 -
 -
Crawler/1.2 - -
 text/..
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
 -
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-maghalesaz.py/r123 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
pywikibot-1newar25.py/r10825 Pywikipediabot/1.0 - -
 application/json
wAPI/1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
pwb/rg2765 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2770 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Peachy MediaWiki Bot API Version 1.0 -
 application/vnd.php.serialized
 text/..
pywikipedia-featured.py/r10846 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
SurakWare MediaWiki Bot/1.0 - -
 application/x-www-form-urlencoded
 text/..
cwikt-interwiki.py/r10843 Pywikipediabot/1.0 - -
 application/json
 application/xml
 -
 text/..
-c/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-zzaddnavebox.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
compat-featured.py/r10843 Pywikipediabot/1.0 - -
 application/json
 -
 text/..
 application/xml
pwb/rg2720 Pywikipediabot/2.0 - -
 application/json
compat-featured.py/r10843 Pywikipediabot/1.0 -
 -
pwb/r2772 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wikify/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
cwikt-interwiki.py/r10854 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pwb/rs3800 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
 application/xml
 application/opensearchdescription+xml
 application/rsd+xml
 -
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en,* -
 image/..
 text/..
logpatrol/rg2764 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-commonscat.py/r10860 Pywikipediabot/1.0 - -
 -
 application/json
 application/xml
COIBot/2.0 - -
 text/..
election/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-zztakhasosi.py/r10694 Pywikipediabot/1.0 -
 -
DotNetWikiBot/3.10 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 application/x-www-form-urlencoded
 text/..
Python27-python.exe/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-zztakhasosi.py/r10694 Pywikipediabot/1.0 - -
 application/json
 text/..
porzucone/rg2725 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2753 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2772 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
MediaWiki::Perl/User:Wdwdbot - -
 application/x-www-form-urlencoded
 application/json
redirect/r2691 Pywikipediabot/2.0 - -
 application/json
ko_missing2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pwb/rg2750 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-newarz.py/r10523 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
cwikt-interwiki.py/r10843 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pwb-reportCatLoops.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-newarz.py/r10523 Pywikipediabot/1.0 - -
 application/json
www.monit24.pl-m24Bot/4.1- - -
 image/..
 -
 text/..
GoogleBot en-us -
 text/..
 -
pywikibot-compat-commonscat.py/r10836 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
Yeobot/1.0 Contact: mail address ) - -
 text/..
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
pywikibot-compat-commonscat.py/r10854 Pywikipediabot/1.0 - -
 -
 application/json
pywikibot-compat-commonscat.py/r10833 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
 application/json
pywikipedia-zzfatranslator.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/x-www-form-urlencoded
pywikipedia-zzredirectyeh.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg2756 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-basicarticledisamblist.py/r10858 Pywikipediabot/1.0 -
 -
cwikt-interwiki.py/r10854 Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/rg2738 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-basicarticledisamblist.py/r10843 Pywikipediabot/1.0 - -
 application/json
compat-tp_project.py/r10385 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
pywikipedia-git-ship2.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikipedia-git-wdph29.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; SputnikBot/2.3) ru,en;q=0.1,*;q=0.01 -
 text/..
 -
pwb/rg2726 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 -
 image/..
pwb/rg2727 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-zzredirectyeh.py/r10694 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
porzucone/rg2720 Pywikipediabot/2.0 - -
 application/json
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
compat-basicarticledisamblist.py/r10843 Pywikipediabot/1.0 -
 -
pwb-klbot_wikdiata_name2.py/r7 Pywikipediabot/1.0 - -
 application/json
 text/..
Biosgraphies Online Crawler (Curl/PHP) mail address - -
 application/json
XLinkBot/1.00 - -
 text/..
 -
pwb/rg2763 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-basicarticledisamblist.py/r10858 Pywikipediabot/1.0 - -
 application/json
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
 -
Pywikibot-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
porzucone/rg2765 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/rg2766 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg2763 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
porzucone/rg2746 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
pywikipedia-git-maghalesaz2.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
porzucone/rg2751 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
MediaWiki::Bot/5.005007 - -
 application/json
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
run_pagefromfile.4/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
dmashine-CleanupTematic.py/r10836 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pywikibot-compat-commonscat.py/r10847 Pywikipediabot/1.0 - -
 application/json
 -
porzucone/rg2761 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 -
 -
pywikibot-compat-commonscat.py/r10828 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/r10841 Pywikipediabot/1.0 - -
 -
 application/json
cwikt-interwiki.py/r10825 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
porzucone/rg2768 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/r10854 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
pwb/rg2767 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-parserfunctioncount.py/r10797 Pywikipediabot/1.0 - -
 application/json
 -
HRoestBot, de-wikipedia using pywikipedia framework -
 text/..
 application/json
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - -
 -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
iw_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Screaming Frog SEO Spider/2.22 - -
 text/..
 -
 image/..
refreshfeeds/rg2770 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
refreshfeeds/rg2753 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-aaa_categorieTesti.py/r10761 Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
 application/x-www-form-urlencoded
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
bot-null.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
Projet Sport-Projetsportliste2.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
pywikibot-compat-commonscat.py/r10853 Pywikipediabot/1.0 - -
 -
 application/json
compat-featured.py/r10860 Pywikipediabot/1.0 - -
 application/json
 application/xml
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
VitalArticleTalkPageUpdate/r2154 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-zzhayati.py/r10694 Pywikipediabot/1.0 - -
 application/json
 application/pdf
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/rg2717 Pywikipediabot/2.0 - -
 application/json
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - -
 application/xml
 text/..
 -
TambonBot Wikibase.NET/0.1 - -
 application/json
 application/x-www-form-urlencoded
YBot/0.1 - -
 application/vnd.php.serialized
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
Full_update/r2554 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pwb/rg2723 Pywikipediabot/2.0 - -
 application/json
pwb/r2234 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Inlibris.com XMLBot/1.0 - -
 text/..
compat-featured.py/r10860 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - -
 application/json
 text/..
compat-featured.py/r10858 Pywikipediabot/1.0 -
 -
compat-featured.py/r10853 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/r10853 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
archiv-erl-bot/r2625 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - -
 text/..
 -
compat-featured.py/r10858 Pywikipediabot/1.0 - -
 application/json
 application/xml
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
refreshfeeds/rg2754 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
compat-featured.py/r10854 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
compat-delete.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
User-Agent Baiduspider - -
 text/..
 image/..
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
 -
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
onidbot - -
 image/..
 text/..
py-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
wymowa/rg2763 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2767 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/r10833 Pywikipediabot/1.0 - -
 application/json
pwb/rs3751 Pywikipediabot/2.0 - -
 application/json
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
pywikibot-1newar25.py/r10846 Pywikipediabot/1.0 - -
 application/json
 -
WPBot 1.0 - -
 text/..
 -
wymowa/rg2761 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
compat-featured.py/r10829 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/rg2752 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/xml
addDescMensch/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-touch.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
redirect/rg2727 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pi-1.py/r10836 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 -
wymowa/rg2726 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
wymowa/rg2765 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
 -
 text/..
 application/json
pywikipedia-replace.py/r10833 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
 text/..
WikiBot/0.1 - -
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
wymowa/rg2727 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzhayati.py/r10694 Pywikipediabot/1.0 -
 -
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 KHTML Chrome/24.0.1290.2 Safari/537.13 Lumibot/2.0 en-US,en;q=0.8 -
 image/..
 text/..
mijnbots-in19.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/xml
 application/x-www-form-urlencoded
 -
 application/json
pywikipedia-git-wdph90.py/r90 Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
niki-bot - -
 text/..
 image/..
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
wymowa/rg2756 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2750 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
gsa-crawler (Enterprise; T3-GMDVWEKT9WSQZ; mail address ) - -
 text/..
 image/..
pywikibot-compat-commonscat.py/r10829 Pywikipediabot/1.0 - -
 application/json
logpatrol/rg2714 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
PureBasic WikiBotFramework - d:User:InkoBot operated by d:User:Inkowik - -
 text/..
gsa-crawler (Enterprise; T4-KW9PXNCULXJSR; mail address ) - -
 text/..
pywikipedia-pwb.py/r10805 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
porzucone/rg2757 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
wikiparser/1 CFNetwork/673.2.1 Darwin/13.1.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
refreshfeeds/rg2725 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/r2693 Pywikipediabot/2.0 - -
 application/json
refreshfeeds/rg2726 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2754 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
SMNutchSpider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia-doi.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
pywikipedia3-welcome.py/r11747 Pywikipediabot/1.0 - -
 application/json
run_replace.7r/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-pwb.py/r10842 Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
 image/..
 text/..
Curious George - www.analyticsseo.com/crawler - -
 text/..
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
HersfoldIRCBot version 1.9.1 - -
 text/..
 -
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
pwb/rg2707 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-wdph29.py/r123 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
User:Bilderbot (derivativeFX) by Luxo on the Toolserver / PHP -
 application/vnd.php.serialized
 text/..
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 -
 application/json
pwb/rg2717 Pywikipediabot/2.0 - -
 application/json
pwb/rg2751 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
refreshfeeds/rg2765 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
zs/rg2720 Pywikipediabot/2.0 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8
 text/..
AdMedia bot - -
 text/..
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
porzucone/rg2714 Pywikipediabot/2.0 - -
 application/json
renamecats/r2631 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
 text/..
TFAdefenderbot/r2303 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
bot-gender.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
Pywikipedia-lonelypages.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 text/..
 application/x-www-form-urlencoded
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 -
 text/..
svn-compat-commonscat.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 text/..
botpywi-analyse_article.py/r10244 Pywikipediabot/1.0 - -
 application/json
 -
 text/..
compat-countexisting.py/r10856 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
refreshfeeds/rg2772 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
 -
 application/json
Python27-pythonw.exe/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
DotNetWikiBot/2.100 (Unix 3.2.0.57; ) - -
 text/..
 application/x-www-form-urlencoded
 application/xml
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 text/..
 image/..
 -
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 application/x-www-form-urlencoded
 text/..
 application/xml
hivaBot/hivaBot-0.2 ( mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
py-new_interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
porzucone/r2691 Pywikipediabot/2.0 - -
 application/json
refreshfeeds/rg2768 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
update-regular-users/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
run_replace.7r.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Just-Crawling 0.1 - -
 text/..
 -
AnomieBOT 1.0 (PUICloser; see [[User:AnomieBOT]]) - -
 application/json
CaBot Script (running on nightshade.toolserver.org) - -
 application/vnd.php.serialized
pywikipedia-commonscat.py/r10694 Pywikipediabot/1.0 -
 -
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
compat-countexisting.py/r10856 Pywikipediabot/1.0 - -
 application/json
 application/xml
logpatrol/rg2770 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
By IdeasBot Framework, Version:1.0.460 - -
 text/..
compat-redirect.py/rs10837 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
 application/xml
pywikibot-compat-pwb.py/r10815 Pywikipediabot/1.0 - -
 application/json
run_replace.7r.3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
replace/rg2756 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
wymowa/rg2707 Pywikipediabot/2.0 - -
 application/json
pywikipedia-imageuncat.py/r11775 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
datetemplates/r2403 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-commonscat.py/r10694 Pywikipediabot/1.0 - -
 application/json
Zing-BottaBot/1.0 - -
 text/..
KatBot bot.nytlabs.com> - -
 text/..
pywikibot-1newar25.py/r10847 Pywikipediabot/1.0 - -
 -
 application/json
GoogleBot-Image/1.0 bg -
 text/..
 image/..
pywikipedia-amb_mehrdadbot.py/r11775 Pywikipediabot/1.0 - -
 application/json
QrobotW - -
 text/..
 image/..
redirect/rg2736 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
Wiki.java 0.30 r183 (OctraBot 3.2.2) - -
 text/..
 -
 application/json
 application/x-www-form-urlencoded
logpatrol/rg2726 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
UAA-UAA.py/r10417 Pywikipediabot/1.0 - -
 application/json
 -
 application/x-www-form-urlencoded
Roka Spider/Nutch-2.2.1 (' mail address ') - -
 text/..
www.monit24.pl-m24Bot/4.1- -
 -
 image/..
TestBot fr -
 text/..
python-wikitools/1.2 (User:LaraBot) -
 application/json
wikinews-interwiki.py/r10856 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
 application/xml
('python-wikitools/1.1.1 (User:BernsteinBot)',) - -
 application/json
AnomieBOT 1.0 (IFDCloser; see [[User:AnomieBOT]]) - -
 application/json
YisouSpider - -
 text/..
Erel Bot - -
 text/..
deadlinks/r2684 Pywikipediabot/2.0 - -
 application/json
wymowa/rg2717 Pywikipediabot/2.0 - -
 application/json
erfgoedbot-categorize_images.py/r11633 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2692 Pywikipediabot/2.0 - -
 application/json
GoogleBot fr -
 text/..
 -
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/537.36 KHTML Chrome/32.0.1700.99 Mobile Safari/537.36 es-ES,es;q=0.8 -
 image/..
 text/..
 -
-c/r2463 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
ProoXiBot fr -
 text/..
pywikipedia-checknewarticles_formaintenance-20130923.py/r11775 Pywikipediabot/1.0 - -
 application/json
redirect/rg2721 Pywikipediabot/2.0 - -
 application/json
pywikipedia-checknewarticles_formaintenance-20130923.py/r11775 Pywikipediabot/1.0 -
 application/json
pywikipedia-featured.py/r10843 Pywikipediabot/1.0 - -
 -
 application/json
 application/x-www-form-urlencoded
deadlinks/rg2743 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 text/..
pywikipedia-zzfawikipatroler.py/r10694 Pywikipediabot/1.0 - -
 application/json
 -
wymowa/rg2725 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/rg2743 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
bitlybot - -
 text/..
 image/..
 -
erfgoedbot-categorize_images.py/r11633 Pywikipediabot/1.0 -
 -
 application/x-www-form-urlencoded
Alex's crawler/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
compat-featured.py/r10833 Pywikipediabot/1.0 -
 -
Mozilla 5.0 (Apibot 0.30b5) - -
 application/vnd.php.serialized
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
svn-compat-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 application/xml
 -
request-daemon/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikibot-compat-pwb.py/r10856 Pywikipediabot/1.0 - -
 application/x-www-form-urlencoded
 application/json
 -
 image/..
commons-substituter.py/r11436 Pywikipediabot/1.0 - -
 application/json
redirect/rg2707 Pywikipediabot/2.0 - -
 application/json
logpatrol/rg2753 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
redirect/rg2732 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
wymowa/rg2720 Pywikipediabot/2.0 - -
 application/json
redirect/rg2734 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
pywikipedia-git-wdph662.py/r90 Pywikipediabot/1.0 - -
 application/json
 application/x-www-form-urlencoded
 -
pywiki-newar25.py/r10843 Pywikipediabot/1.0 - -
 -
 application/json
CoBot/1.0 (Olivaw) - -
 application/json
refreshfeeds/rg2720 Pywikipediabot/2.0 - -
 application/json
BotProoXi fr -
 text/..
 -
lgbtbot.com [ZSEBOT] - -
 text/..
rfbotstatus/r2631 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
bot-fi-decl.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
redirect/r2695 Pywikipediabot/2.0 - -
 application/json
zd/rg2720 Pywikipediabot/2.0 - -
 application/json
compat-newinterwiki-ar.py/r10334 Pywikipediabot/1.0 - -
 application/json
PercolateCrawler/4 ( mail address ) - -
 text/..
 image/..
 -
pywikipedia-pwb.py/r10805 Pywikipediabot/1.0 -
 application/x-www-form-urlencoded
 -
 text/..
logpatrol/rg2746 Pywikipediabot/2.0 - -
 application/x-www-form-urlencoded
 application/json
redirect/rg2748 Pywikipediabot/2.0 - -
 application/json
 application/x-www-form-urlencoded
36000.33total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Wed, Mar 12, 2014 12:15
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers