Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Aug 2013 - 31 Aug 2013 (last 12 months)
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

Warning: all recent Wikimedia traffic analysis reports have been generated from old scripts.

The scripts are orphaned, and have not been maintained for at least 6 months. Many bugs are considerably older.
Known Bugzilla bugs: 46190, 46191, 46195, 46201, 46205, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 46289

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 134,292,680 page requests (mime type text/html only!) per day are considered crawler requests, out of 603,565,230 external requests, which is 22.2%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk9-hrd) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy4-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile8-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~appcodes-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~scroll-down-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6001-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy5070-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~ddbrite-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk8-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6002-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~i-am-proxy-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk5-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython7-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~cgiproxy5-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~bereturning-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourbudgets-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf702-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~response-out-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~usersaccess-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6006-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~front-pages-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~tutorialses-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf605-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~flyproxy0-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy8-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~goodersearch-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~for-language-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~howtodoita-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython9-hrd) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf706-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~guidesites-hrd) - -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..Mozilla/5.0 (X11; Linux x86_64; zh_CN) KHTML/4.10.5 Konqueror/4.10 AppEngine-Google; (url; appid: s~42overwall) zh-CN,en-US;q=0.9,en;q=0.8 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~espanatiki) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~news-world-me) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kittehserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~amort141) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps004) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~secure-facebook-ranney) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kuryproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: srvr1969) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: time-ghost) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: unlimitedbrowsing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~theunblock) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-phpproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ape-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kaveriselvaraj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf002) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: easypox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxworx) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nmimsforti) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 www.google.com/coop/cse/creftext/..PageFetcher-Google-CoOp;((url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~batshiitinsane) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mrjosiaht) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: javahavens) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kurizogeorge) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usingomega) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thepeterdeutsch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~fox-eye) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d-accord-proxyserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pxdrill) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy7) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: moelonepyaeshan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sauravprakash-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 6.1; zh-CN; rv:1.9.2.2) AppEngine-Google; (url; appid: fwall-w15) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mboharsik) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq5) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zagrobelnyprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pleasesendmeto) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps002) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~strakios) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jptaravellahighschool) - -
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: s~libmuteki2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: shoby-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jompr0xy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxynaungnaung) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tom-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mistakeproxyarea) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crackernew-1234-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kickermonkpx) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix) - -
 code.google.com/appenginetext/..Acre/dev/53:914 factchecker.freebase-refinery.appspot.com AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~no-firewall) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nothing-unusual) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~zmeomzm) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: failboatweb) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hydraroxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kappasan-mirror) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vietwebdisk) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: caineserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sirpats) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: burmansearch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: inviofx-prxs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bypass-filter) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy2bay) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vi-mobile) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cyberdudeproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fadaanan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~pro-xy007) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chrissygallacherr) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: randohnson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: grandpotato18) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kylesumrallproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~simolifet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ipreflector2) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 KHTML Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (url; appid: s~feedly-social) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ageryder) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nallanikrithika) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: this-is-not-what-u-think) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cmd-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tycko-gc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: arturoproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ownsterserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl4) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kauflog) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ahsanovic) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jttm-server-prox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: miniden-proxi) - -
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: puthiyathiravidan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tyangbanga) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: argim-free) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: trabserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: krittproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: school-x) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadl6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~italiatiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: djsk-moon) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: muramasa1980) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-eye-ear) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tinkernutsearch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tacomansterprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: prixi2010) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ilovethemothernature) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: funnyinternetwebsite) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~app3123ak) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps001) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jasonlimcheewai) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-data) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mitchblows) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: navnode) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beholdonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-myanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 KHTML Chrome/28.0.1500.95 Safari/537.36 AppEngine-Google; (url; appid: s~cwek-gate1) zh,zh-CN;q=0.8,zh-TW;q=0.6,en;q=0.4,ja;q=0.2 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: he4lproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gizmo-jumpjet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: xbmacx) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lovelamp1988) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thomproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cdeskinsp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: calymirror) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: matbuot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demowaiy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nstahl2124) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~herpity-derpity) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline7) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kyaysarlay) - -
 code.google.com/appenginetext/..Acre/dev/59:906 ubiquity.freebaseapps.com AppEngine-Google; (url; appid: s~freebaseapps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hbypasser) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxies-list2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~pey3be) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadl4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boweiprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nikita-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cymanconsole) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: inetbrowse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: korvas-sux) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nethinguwnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khrixy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps005) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~4youtech22) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: vi-mobile) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sl-santabanta) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: the-snyke) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tre-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kha1989led) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) - -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
news
 www.news.net/text/..News.Net/1.1 (url; mail address ) BasedOnRawPHP - -
 www.news.net/application/jsonNews.Net/1.1 (url; mail address ) BasedOnRawPHP - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/jsonGoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ja-JP,ja -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pp8hh4) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
msn
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)1 readers - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) - -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/4.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot.FreshPages/0.1; url) - -
 ahrefs.com/application/xmlAhrefsBot.Feeds v0.1; url - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/text/..WikipediaApp_svc/1.5.13 (url; mail address ) com.orange.mmp/1.7.0 - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) zh-cn;q=1.0, zh-tw;q=0.8, en;q=0.5, *;q=0.1 -
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) - -
 help.naver.com/robots/image/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ko,ja,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.1 (NHN Corp.; url) vi_VN,vi;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url) ja,en;q=0.5 -
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) NOT Firefox/3.5 en-us,en;q=0.5 -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2 - -
 help.yahoo.com/help/us/ysearch/slurpapplication/xmlMozilla/5.0 (compatible; Yahoo! Slurp;url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en-us,en;q=0.5 -
143
 173.13.143.74/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url) - -
 173.13.143.74/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
soso
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Mozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmapplication/jsonMozilla/5.0 (compatible; Sosospider/2.0; url) - -
 help.soso.com/webspider.htmimage/..Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htmtext/..Mozilla/5.0(compatible;Sosospider/2.0;url) zh-cn,zh-hk,zh-tw,en-us -
 help.soso.com/webspider.htm-Mozilla/5.0 (compatible; Sosospider/2.0; url) zh-cn,zh-hk,zh-tw,en-us -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,ua;q=0.7,by;q=0.7,*;q=0.1 -
sblog
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url) - -
 fulltext.sblog.cz/-SeznamBot/3.0 (url) cs -
 fulltext.sblog.cz/screenshot/application/jsonMozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.1-test2; url) * -
 fulltext.sblog.cz/-Mozilla/5.0 (compatible; SeznamBot/3.1-test1; url) cs -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url) zh-cn -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 2.6.32-50-server; java 1.6.0_27; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-27-generic; java 1.7.0_25; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86_64 Mac OS X 10.7.3; java 1.6.0_35; Europe/de) url - -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (KULeuven_research/global; i386 Linux 2.6.32-36.onelab.x86_64; java 1.7.0_09; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal/global; amd64 Linux 3.9-1-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-27-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; US/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-358.6.2.el6.centos.plus.i686; java 1.6.0_24; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-27-generic; java 1.7.0_25; Pacific/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (webportal-global; amd64 Linux 3.9.4-030904-generic; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (KULeuven_research/global; i386 Linux 2.6.32-36.onelab.i686; java 1.7.0_09; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; GMT01:00/it) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-50-server; java 1.6.0_26; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-29-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows Server 2008 6.0; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-51-generic; java 1.6.0_27; Atlantic/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-48-generic; java 1.8.0-ea; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.10-2-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-48-generic; java 1.8.0-ea; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.6.0_41; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-042stab078.27; java 1.6.0_27; Etc/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.8.0-27-generic; java 1.7.0_25; Europe/nl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-50-generic; java 1.6.0_27; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; i386 Linux 2.6.32-358.14.1.el6.centos.plus.i686; java 1.6.0_24; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.5.0-23-generic; java 1.7.0_25; Asia/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-50-generic; java 1.6.0_27; Europe/es) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (allip/any; amd64 Linux 3.8.13-xxxx-grs-ipv6-64; java 1.6.0_27; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.9-1-amd64; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) zh-cn;q=0.8, *;q=0.5 -
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url) - -
 shoulu.jike.com/spider.htmlapplication/jsonMozilla/5.0 (compatible; JikeSpider; url) - -
wordpress
 coyunturainternacionalamerica.wordpress.comtext/..WordPress/3.7-alpha-24954; url - -
 radishmag.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 radishmag.wordpress.comtext/..WordPress/3.6; url - -
 robertlindsay.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 michaelruark.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 managra.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 josefboberg.wordpress.comtext/..WordPress/3.7-alpha-24954; url - -
 at37.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 klausgauger.wordpress.comtext/..WordPress/3.7-alpha-24954; url - -
 peoplesadvocacycouncil.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 greatriversofhope.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 josefboberg.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 klausgauger.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 tsjok45.wordpress.comtext/..WordPress/3.7-alpha-25000; url - -
 radishmag.wordpress.comtext/..WordPress/3.7-alpha-24954; url - -
acoon
 www.acoon.de/robot.asptext/..Mozilla/5.0 (compatible; AcoonBot/4.11.1; url) - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.4; url) en -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
crossref
 alm.labs.crossref.orgapplication/jsonArticle Level Metrics - url - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.21.0 url - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBottext/..OrlodrimBot/1.0 (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.20.0 url - -
nodejs
 nodejs.orgtext/..NodeJsWikipediaReader/0.1 (url; mail address ) BasedOnNodeJs/0.8.22 - -
pchome
 www.pchome.com.tw/pchomebot.htmtext/..Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
 www.pchome.com.tw/pchomebot.htm-Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
zipcode
 zipcode.ustext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
flipboard
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.2; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
webmeup
 webmeup.com/crawler.htmltext/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url) - -
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url) - -
 toolserver.org/~dispenser/text/..CacheThumbs/1.2 (url) -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02 - -
 toolserver.org/~dispenser/image/..CacheThumbs/1.2 (url) -
bibalex
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 goo.gl/7y4SXimage/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
 goo.gl/7y4SXtext/..GoogleProducer; (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..ichiro/3.0 (url) - -
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url) - -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url) - -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
plos
 alm.plos.orgapplication/jsonPLOS Article Level Metrics - url - -
 alm2-iad.plos.orgapplication/jsonArticle-Level Metrics - url - -
 alm2-iad.plos.orgapplication/jsonArticle Level Metrics - url - -
in
 www.m-culture.in.thtext/..m-culture.in.th (url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address ar -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address es-es -
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.85; url) Gecko/2008032620 - -
wikidict
 www.wikidict.detext/..url - -
zookabot
 zookabot.comtext/..Zookabot/2.5;url - -
yunyun
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/SiteInfo.php?r=abouttext/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/SiteInfo.php?r=aboutimage/..Mozilla/5.0 (compatible; YRSpider; url) zh-cn;q=0.8, *;q=0.5 -
 www.yunyun.com/spider.htmltext/..Mozilla/5.0 (compatible; YYSpider; url) - -
 www.yunyun.com/SiteInfo.php?r=abouttext/..Mozilla/5.0 (compatible; YRSpider; url) - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
sourceforge
 linkchecker.sourceforge.net/text/..LinkChecker/7.4 (url) - -
 linkchecker.sourceforge.net/image/..LinkChecker/7.4 (url) - -
scrapy
 scrapy.orgtext/..Scrapy/0.16.4 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
 scrapy.orgtext/..Scrapy/0.18.0 (url) lt -
 scrapy.orgtext/..Scrapy/0.18.0 (url) ceb -
 scrapy.orgtext/..Scrapy/0.18.0 (url) am -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) - -
example
 alm.example.orgapplication/jsonArticle Level Metrics - url - -
 example.com/MyCoolTool/text/..User-Agent: MyCoolTool/1.1 (url; mail address ) - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.com/MyCoolTool/text/..MyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
import
 import.iotext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.ninjal.ac.jp/corpus_center/ulc/crawl-entext/..Mozilla/5.0 (compatible; heritrix/3.1.1 url) - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.html-Feedly/1.0 (url; like FeedFetcher-Google) - -
wikimedia
 tools.wikimedia.de/~daniel/text/..WikiSense (url) - -
 commons.wikimedia.org/wiki/User:Thumbnails_Check_Botimage/..Thumbnails_Check_Bot/0.1 (url; beta) - -
 meta.wikimedia.org/wiki/User:Tietewtext/..Cheebot/0.5.7 (url) - -
 tools.wikimedia.de/~para/GeoCommons/text/..url - -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
zeerch
 zx1.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; ZXBOT-ZX2; url) - -
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; ZEERCHBOT; url) - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
automoveisnaweb
 automoveisnaweb.com.br/uk1text/..WordPress/3.6; url - -
 automoveisnaweb.com.br/uk3text/..WordPress/3.6; url - -
 automoveisnaweb.com.br/uk2text/..WordPress/3.6; url - -
psu
 citeseerx.ist.psu.edutext/..citeseerxbot (compatible; heritrix/1.14.4 url) - -
 citeseerx.ist.psu.eduimage/..citeseerxbot (compatible; heritrix/1.14.4 url) - -
github
 wummel.github.com/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.4; url) - -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8 - -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
tineye
 tineye.com/crawler.htmlapplication/jsonTinEye/1.1 (url) - -
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url) - -
 enwp.org/User:Hellknowz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) hu -
xbmc
 www.xbmc.orgimage/..XBMC/12.2 Git:20130502-32b1a5e (Windows NT 6.1;WOW64;Win64;x64; url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
profound
 www.profound.net/urlappendbot.htmltext/..Mozilla/5.0 (compatible; URLAppendBot/1.0; url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
politicalvelcraft
 politicalvelcraft.orgtext/..WordPress/3.7-alpha-24954; url - -
 politicalvelcraft.orgtext/..WordPress/3.6; url - -
 politicalvelcraft.orgtext/..WordPress/3.7-alpha-25000; url - -
 politicalvelcraft.orgtext/..WordPress/3.6-RC2-24937; url - -
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
msai
 www.msai.in/uaprof/micromax/X455.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X455.xmltext/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmlimage/..url en,hi -
 www.msai.in/uaprof/micromax/X1i_Extra.xmltext/..url en,hi -
dragonoff
 www.dragonoff.com/php/dev/WikifySong.phpapplication/vnd.php.serializedurl - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) - -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url) - -
trendiction
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url) - -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
rifanmuazin
 trends.rifanmuazin.comimage/..WordPress/3.6; url - -
 trends.rifanmuazin.comtext/..WordPress/3.6; url - -
gamedipper
 www.gamedipper.comapplication/jsongamedipper.com bot (url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
linkdex
 www.linkdex.com/about/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
tivine
 tivine.com/application/jsonTivine_0.01 (url; mail address ) Test Developments - -
go
 kc.nict.go.jp/project1/crawl-ja.htmltext/..ICC-Crawler (Mozilla-compatible; mail address ; url) ja -
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20121013.132750 url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
rebelmouse
 rebelmouse.comimage/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
 rebelmouse.comtext/..RebelMouse/0.1 Mozilla/5.0 (compatible; url) Gecko/20100101 Firefox/7.0.1 en-us,en;q=0.5 -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; Monitis - premium monitoring service; url) - -
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url) - -
metamagazine
 metamagazine.comtext/..WordPress/3.5.2; url - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
wiktionary
 en.wiktionary.org/wiki/User:Rukhabotapplication/jsonRukhabot/0.1 (url) - -
axtaris
 axtaris.orgtext/..Hello from Axtaris-Crawler! (url) - -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, url) - -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
js-kit
 js-kit.com/text/..JS-Kit URL Resolver, url - -
readability
 readability.com/about/text/..Readability/b5bf8e - url - -
 readability.com/about/text/..Readability/38b3ab - url - -
 readability.com/about/text/..Readability/73c8c6 - url - -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url) - -
superfeedr
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.com-Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
turnitin
 www.turnitin.com/robot/crawlerinfo.htmltext/..TurnitinBot/2.1 (url) - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.8 (url) - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
sf
 liferea.sf.net/application/xmlLiferea/1.8.3 (Linux; fr_FR.UTF-8; url) - -
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url) en-us,en;q=0.5 -
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url) en-us,en;q=0.5 -
feuerwehr-sasbach
 www.feuerwehr-sasbach.detext/..MyCoolTool/1.1 (url; mail address - -
176
 118.139.176.95/wordpress/rss/test.phptext/..MyBot/1.0 (url) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
mjotiu
semrush
 www.semrush.com/bot.htmltext/..Mozilla/5.0 (compatible; SemrushBot/0.96.4; url) - -
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url) - -
hvacrepairmedic
 denver.hvacrepairmedic.comtext/..WordPress/3.5.2; url - -
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/4.0.0 (Mac OS X; url; gzip-happy) en-us -
kalooga
 kalooga.com/crawlerimage/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
 kalooga.com/crawlertext/..Mozilla/5.0 (compatible; KaloogaBot; url) - -
zapbot
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url) - -
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url) - -
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url) - -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
 wiki.creativecommons.org/DiscoverEdtext/..My Nutch Spider/Nutch-1.7 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url) en-us,en;q=0.5 -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) - -
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1 en-US -
localhost
ibis
 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url) - -
 ibis.ne.jp/browser/about.htmltext/..Mozilla/4.0 (compatible; ibisBrowser; url) - -
pride-news
 pride-news.nettext/..WordPress/3.6; url - -
161330.54999998total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
Peachy MediaWiki Bot API Version 2.0 (alpha 4) - -
 application/vnd.php.serialized
 text/..
GoogleBot-Image/1.0 - -
 image/..
 text/..
 -
 application/pdf
 application/ogg
redirect/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
php wikibot classes - -
 application/vnd.php.serialized
 text/..
 -
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
 -
wikiwix-bot-3.0 - -
 text/..
 -
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
 text/..
Addframe Mediawiki bot - -
 application/vnd.php.serialized
 text/..
 -
LinkParser/2.0 - -
 text/..
 -
PythonWikipediaBot/1.0 - -
 application/json
 application/xml
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
GoogleBot-Image/1.0 - -
 text/..
 image/..
 -
 application/json
Peachy MediaWiki Bot API Version 2.0 (alpha 5) - -
 application/vnd.php.serialized
 text/..
tigerbot - -
 application/json
 text/..
spider - -
 text/..
 application/vnd.php.serialized
 application/json
 image/..
www.integromedb.org/Crawler - -
 text/..
 -
 application/xml
 application/json
pywikipedia-wikidata_missinglabel_add.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/r2872 Pywikipediabot/2.0 - -
 application/json
redirect/r2871 Pywikipediabot/2.0 - -
 application/json
ClueBot/1.1 - -
 application/vnd.php.serialized
 -
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
 -
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 -
YisouSpider - -
 text/..
 -
 application/ogg
redirect/r2897 Pywikipediabot/2.0 - -
 application/json
redirect/r2895 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; Ezooms/1.0; mail address ) - -
 text/..
 application/json
 image/..
 audio/webm
 video/webm
Bot: mail address - -
 text/..
WikidataBot framework - -
 application/json
 text/..
pywikipedia-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
maj_articles_recents/r11700 Pywikipediabot/2.0 - -
 application/json
ClueBot/2.0 - -
 application/vnd.php.serialized
pywikipedia-redirect.py/r11168 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia2-redirect.py/r11775 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Bot/5.005006 - -
 application/json
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
welcome-welcome-w-bn-core.py/r11692 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.101 (Unix 3.2.0.51; ) - -
 text/..
redirect/r2882 Pywikipediabot/2.0 - -
 application/json
redirect/r2874 Pywikipediabot/2.0 - -
 application/json
maj_articles_recents/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
redirect/r2890 Pywikipediabot/2.0 - -
 application/json
redirect/r2891 Pywikipediabot/2.0 - -
 application/json
redirect/r2879 Pywikipediabot/2.0 - -
 application/json
DigitalsmithsBot - -
 text/..
redirect/r2896 Pywikipediabot/2.0 - -
 application/json
Sogou Web Spider zh-cn -
 text/..
 -
CorenSearchBot/1.7 en libwww-perl/6.04 - -
 text/..
My Nutch Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 application/ogg
 image/..
 video/webm
MediaWiki::Bot/3.2.6 - -
 application/json
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
itemfix/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
itemfix/r1909 Pywikipediabot/2.0 - -
 application/json
Peachy MediaWiki Bot API Version 2.0 (alpha 3) - -
 application/vnd.php.serialized
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
Mozilla/5.0 (compatible; SearchBot) - -
 text/..
pywikipedia-welcome.py/r11533 Pywikipediabot/1.0 - -
 application/json
p-welcome-d-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
p-welcome-w-bn-core.py/r10872 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
 image/..
 -
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
Sogou web spider/4.0 zh-cn -
 text/..
 -
moje-wymowa.py/r11483 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot) - -
 text/..
Tawbot (public svn release; plwiki) - -
 text/..
360spider-image - -
 image/..
 text/..
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_disamb_dp_labelonly.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-interwiki.py/r11655 Pywikipediabot/1.0 - -
 application/xml
 application/json
bot: fr-anal - -
 application/json
pywikipedia-featured.py/r11747 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia3-MakeScoreTable_test2.py/r11448 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
pywikipedia-git-wdph38.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-fr.wikt.RemiseEnForme.py/r11692 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
Semantix Bot 0.1 - -
 text/..
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address ) - -
 text/..
 -
 application/xml
moje-porzucone.py/r11483 Pywikipediabot/1.0 - -
 application/json
GermCrawler - -
 application/json
 text/..
 application/xml
botjagwar-crawlrandom.py/r11215 Pywikipediabot/1.0 - -
 application/json
Test Webbot - -
 text/..
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
pywikipedia-featured.py/r11674 Pywikipediabot/1.0 - -
 application/json
 application/xml
WikiBot/0.1 - -
 text/..
 image/..
milog_bot/1.0 ( mail address ) - -
 text/..
Wikibot/2.1.1 CFNetwork/609.1.4 Darwin/13.0.0 en-us -
 image/..
 text/..
COIBot/1.00 - -
 text/..
botjagwar-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
maj_articles_recents/r1995 Pywikipediabot/2.0 - -
 application/json
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
SAS Web Crawler - -
 text/..
 -
 image/..
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
WikiCatResearchBot ( mail address ) - -
 text/..
pywikipedia-dikantenyvaovao.py/r11215 Pywikipediabot/1.0 - -
 application/json
matilda/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
py-welcome.py/r11775 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 5.10.0.0; ) - -
 text/..
 application/xml
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
compat-commonscat.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
new_interwiki/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
Webbull-Bot - -
 text/..
 -
Jbot - -
 text/..
Junut ping bot - -
 text/..
 -
DotNetWikiBot/2.100 (Unix 3.2.0.48; ) - -
 text/..
wikidata-sitelinks.py/r11618 Pywikipediabot/1.0 - -
 application/json
COIBot/2.0 - -
 text/..
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-dobrodoslice.py/r11692 Pywikipediabot/1.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
py-newar5.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
MyCuteBot/0.1 - -
 text/..
 application/json
 application/vnd.php.serialized
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 - -
 application/json
pywikipedia-archivebot.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
peoplecheck.de Crawler - -
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.51; ) - -
 text/..
pywikipedia-28main.build89.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
py-hamsang.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
TicketsBot/0.1 - -
 text/..
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
pywikirish-welcome.py/r11215 Pywikipediabot/1.0 - -
 application/json
pyadmin-autoredirect.py/r11714 Pywikipediabot/1.0 - -
 application/json
 text/..
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
matilda/r11686 Pywikipediabot/2.0 - -
 application/json
www.monit24.pl-m24Bot/4.1- - -
 image/..
 text/..
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
newser-wikinewser2.py/r11775 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 KHTML PhantomJS/1.9.0 Safari/534.34 CasperJS/1.0.2 ve-dirtydiffbot en,* -
 image/..
 text/..
 application/json
DeletionBot-deletion.py/r11525 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
Cyberfox Spider ru, en -
 text/..
 -
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
NikoBot1.0 - -
 text/..
 image/..
dtSearchSpider - -
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider - -
 text/..
 application/json
 application/xml
pywikipedia-interwiki.py/r11775 Pywikipediabot/1.0 - -
 application/xml
 application/json
 text/..
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
bot-linkalt.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
 -
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikipedia-wikidata_coordinaten_infobox.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
wd_import/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
 text/..
EarwigBot/0.2.dev.git204cbce4 (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
 text/..
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
HTMLParser/2.0 - -
 text/..
HRoestBot, de-wikipedia using pywikipedia framework - -
 text/..
 application/json
BotReversor-BotReversor.py/r11775 Pywikipediabot/1.0 - -
 application/json
ht2p-bot/v1 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
pywikipedia-git-wdph22.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
SiocWikiBot/1.0 - -
 application/vnd.php.serialized
 text/..
wikiparser/1 CFNetwork/596.4.3 Darwin/12.4.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
pywikipedia-17-progreso1911.py/r11775 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; UnisterBot; mail address ) - -
 text/..
 application/json
pywikipedia-importwdwi.py/r11775 Pywikipediabot/1.0 - -
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
wikidata-sitelinks.py/r11176 Pywikipediabot/1.0 - -
 application/json
JBot - -
 application/json
aleph-avbot.py/r11368 Pywikipediabot/1.0 - -
 application/json
 text/..
SurakWare MediaWiki Bot/1.0 - -
 text/..
pywikipedia-welcome.py/r11027 Pywikipediabot/1.0 - -
 application/json
ASEBot - -
 text/..
pywikipedia-radeh.py/r11762 Pywikipediabot/1.0 - -
 application/json
merge/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
XLinkBot/1.00 - -
 text/..
JavaCrawler/1.1 - -
 text/..
logpatrol/r11761 Pywikipediabot/2.0 - -
 application/json
commons-renamecats.py/r11674 Pywikipediabot/1.0 - -
 application/json
Soundkiosk Relation-Crawler (Version 1.0; soundkiosk.de) - -
 application/xml
 text/..
FAST Search Web Crawler 14.0.0291.0000 - -
 text/..
 -
HosiryuhosiBot IRC-RecentChanges Checker ja -
 text/..
g13bot_tools-g13_db_maintenance.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
User-Agent Baiduspider - -
 text/..
 image/..
 -
Keybot Translation-Search-Machine - -
 text/..
pywikipedia-wikidata_coordinaten_coord_dec_inline.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-chrwikt.py/r11215 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_setproperty.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywikipedia-pwb.py/r11762 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
 image/..
Mozilla/5.0 (compatible; MyBot/1.0;) - -
 application/json
 text/..
pywikipedia-wikidata_coordinaten_coor_title_dec.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) - -
 text/..
Empedia Bot - -
 text/..
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
commons-imagerecat.py/r11028 Pywikipediabot/1.0 - -
 application/json
Desktop-test.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
pywikipedia-git-wdph23.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-importwdwi.py/r11746 Pywikipediabot/1.0 - -
 application/json
Ruby, Crawler - -
 text/..
unknown, 2a00:4b00:13d:200::129d Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
ReaperBot/2.0.0 (incompatible-notwebbrowser:robot:exclusion-noncompliant) bot> - -
 text/..
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.7) - -
 text/..
2586386/r11657 Pywikipediabot/2.0 - -
 application/json
maj_articles_recents/r1988 Pywikipediabot/2.0 - -
 application/json
pywikipedia-WPP.py/r11746 Pywikipediabot/1.0 - -
 application/json
 application/xml
new2-zzdi1.py/r11452 Pywikipediabot/1.0 - -
 application/json
 text/..
 application/xml
unknown, 2a00:4b00:13d:200::129b Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
py-linktranslator.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_coordinaten_NL.py/r11200 Pywikipediabot/1.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129f Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
pywikipedia-zzlangcat.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-replace.py/r11522 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
unknown, 2a00:4b00:13d:200::129a Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
FAST Search Web Crawler 14.0.0325.0000 - -
 text/..
 application/ogg
 -
DotNetWikiBot/2.100 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
unknown, 2a00:4b00:13d:200::129c Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
AnonBot - -
 text/..
pywikipedia-basicarticledisamblist.py/r11747 Pywikipediabot/1.0 - -
 application/json
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Global-Crawler/Nutch-1.4 (Search platform crawler) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia-welcome-custom.py/r11354 Pywikipediabot/1.0 - -
 application/json
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
theWxitBot/0.1 - -
 application/json
 image/..
bot-null.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
pywikipedia-wd_move.py/r11775 Pywikipediabot/1.0 - -
 application/json
2508035/r11657 Pywikipediabot/2.0 - -
 application/json
/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
unknown, 2a00:4b00:13d:200::129e Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
rg/r11515 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4 GoogleBot/2.1 - -
 image/..
 text/..
pywikipedia-hotarticle.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdph25.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.52; ) - -
 text/..
dabtouch/r11657 Pywikipediabot/2.0 - -
 application/json
Zing-BottaBot/2.0 - -
 text/..
pywikipedia-redirect.py/r11385 Pywikipediabot/1.0 - -
 application/json
 text/..
pyreza-zzredirectyeh.py/r11514 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
WPendata-image_sur_enNaissance.py/r10533 Pywikipediabot/1.0 - -
 application/json
GoogleBot - -
 text/..
artists/r11515 Pywikipediabot/2.0 - -
 application/json
bot-VM-auto-erl.py/r11775 Pywikipediabot/1.0 - -
 application/json
Wikipediabot-wikidata_gndcheck.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
YBot/0.1 - -
 application/vnd.php.serialized
request-daemon/r11515 Pywikipediabot/2.0 - -
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
 -
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
pywikipedia-replace.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
 application/xml
bot-VM-auto-erl.py/r11602 Pywikipediabot/1.0 - -
 application/json
pywikilab-llista_mils_2013.py/r11775 Pywikipediabot/1.0 - -
 application/json
GoogleBot en-us,en;q=0.5 -
 text/..
pywikipedia-infobox.py/r11549 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wdph22.py/r11746 Pywikipediabot/1.0 - -
 application/json
AdMedia bot - -
 text/..
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Local Site Parser 1.0 en-us,en;q=0.5 -
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider zh-CN -
 text/..
 image/..
 application/ogg
 -
datetemplates/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
bot - -
 application/json
 text/..
BotReversor-BotReversor.py/r11707 Pywikipediabot/1.0 - -
 application/json
msnbot - -
 text/..
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
commons-commons-commands.py/r11775 Pywikipediabot/1.0 - -
 application/json
 text/..
Joycrawler Robot - -
 text/..
botjagwar-tasks.py/r11215 Pywikipediabot/1.0 - -
 application/json
wddivisioniamministrative/r11473 Pywikipediabot/2.0 - -
 application/json
pywikipedia-welcome-custom.py/r11775 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
python-wikitools/1.2 (User:LaraBot) - -
 application/json
Big Data Permaculture Localmode Search Spider/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
wikiscore-MakeScoreTable.py/r11549 Pywikipediabot/1.0 - -
 application/json
Mozilla/4.0 (compatible; MT search portal spider/3.0; mail address )" - -
 application/xml
 text/..
gsa-crawler (Enterprise; T2-NXAZZJSZXNWJB; mail address ) - -
 text/..
 -
Screaming Frog SEO Spider/2.20 - -
 text/..
 image/..
Python27-pythonw.exe/r10533 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-basicsbz.py/r11747 Pywikipediabot/1.0 - -
 application/json
commons-nulledit.py/r11674 Pywikipediabot/1.0 - -
 application/json
 text/..
pywikipedia-git-wdph56.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
claimit/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
update-task-categories/r11515 Pywikipediabot/2.0 - -
 application/json
AnomieBOT 1.0 (CHUUClerk; see [[User:AnomieBOT]]) - -
 application/json
 text/..
commons-substituter.py/r11436 Pywikipediabot/1.0 - -
 application/json
Projet IW-PIW2.py/r10533 Pywikipediabot/1.0 - -
 application/json
opentask-opentasks.py/r10326 (pywikibot/__init__.py) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.96 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
MediaWiki::Bot/v3.4.2 - -
 application/json
Inlibris.com XMLBot/1.0 - -
 text/..
pynik-checkimages.py/r11775 Pywikipediabot/1.0 - -
 application/json
pywikipedia-interwikibot2.py/r11597 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.102 (Unix 3.0.0.12; ) - -
 text/..
 application/xml
pywikipedia-fixing_redirects (4).py/r11650 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-com_cat.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
HopperBot - -
 image/..
 text/..
 application/ogg
MoodleBot/1.0 - -
 application/xml
 application/vnd.php.serialized
 text/..
 -
 image/..
python-wikitools/1.2 (User:Mr.Z-bot) - -
 application/json
Spider - -
 application/json
 text/..
MediaWiki::Bot 3.1.5 - -
 application/json
Mozilla/5.0 (Bgbot 0.5) - -
 text/..
pywikipedia-git-wdph25.py/r6 Pywikipediabot/1.0 - -
 application/json
Crawler/1.2 - -
 text/..
trunk-checkimages.py/r11781 Pywikipediabot/1.0 - -
 application/json
pywikipedia-run_replace.5.py/r11641 Pywikipediabot/1.0 - -
 application/json
 application/xml
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
botpywi-analyse_article.py/r11508 Pywikipediabot/1.0 - -
 application/json
Bot - -
 text/..
 image/..
maj_articles_recents/r1984 Pywikipediabot/2.0 - -
 application/json
33546.0800000001total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, Oct 4, 2013 15:06
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers