Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Aug 2014 - 31 Aug 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 157,313,710 page requests (mime type text/html only!) per day are considered crawler requests, out of 641,618,870 external requests, which is 24.5%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~scroll-down-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6001-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile8-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy4-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pyproxy0-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython7-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pyproxy4-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxycgi1-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6002-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf605-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~usethisproxy-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~guidesites-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk8-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk9-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf702-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~adsettings-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~appcodes-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~lookingforwe-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile2-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~payonlyproxy-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~overviewa-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf717-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~tutorialses-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython2-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython9-hrd) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~goodersearch-hrd) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~ddbrite-hrd) - -
 www.bing.com/bingbot.htmimage/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~i-am-proxy-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk5-hrd) - -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaxminds) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: carlos-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: student-pupil-help) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jakestokes) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: img-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ownsterserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: predictionwrong) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: islom-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: iprithi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jactinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sl-santabanta) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thepeterdeutsch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jerproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usingomega) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azzaziprxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: clh-sw) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khsprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp4) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aboytes13tls) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vi-mobile) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-data) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: encalada-proxy-one) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~psjimi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: izzy-queenofthefae) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lavprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imabouncytigger) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaxminds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pramod-ajgaonkar-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zmomyanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: techjawaproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sonydonin) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~amort141) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mahshighschool) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: vi-mobile) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: maltingsproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~net-people) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: goeverysite) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~lime-free) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thegatetoweb) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~simolifet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mboharsik) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: myproxywx) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kakuro69) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appengineapplication/jsonwikipedia (https://github.com/goldsmith/Wikipedia/) AppEngine-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tofi2k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bie99miracle) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kha1989led) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nation4india) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webduplicate) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~skyrideraj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: argim-free) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyproxy2884) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~devonispretty) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-mastered) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gregatr0n) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: fahad-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~agilsikluk3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~techdatproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demowaiy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~aurora-prox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tycko-gc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaproxyserver) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yuricamara) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: caineserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ninjallamastuff) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kerouanen) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~aman11dh) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ritroxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nashma4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tek-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: batamsearch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~secretinternet2011-2012) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nishanthreddysingham) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boweiprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jayjudah1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: korvas-sux) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: shoby-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~jaydenpwns1) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/27.0.1453.110 Safari/537.36 AppEngine-Google; (url; appid: s~livescorefeed-hrd) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ajprxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitalhunch-freenet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thinkersearch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: guoxiangming) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mapremiereapplication) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: trabserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~pridvidhar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go6internet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ihack-tunnel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davidgotmoney50) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~batshiitinsane) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: its-a-tr) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azshaderserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~getwebaddress) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-eye-ear) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: red-arg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: time-ghost) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: unlimitedbrowsing) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; drawings; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bakapouetprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: funnyinternetwebsite) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crowdsurfer100) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pleasesendmeto) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujean) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mitchblows) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxitalia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~certificate-check-403) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nicoflysurf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rxrwng) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~win8wale) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~get-access) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mistakeproxyarea) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nuraiitj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thanhtut88) - -
 code.google.com/appenginetext/..Mozilla/3.0 (compatible) AppEngine-Google; (url; appid: s~tonybashworld) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~strakios) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy7) - -
 code.google.com/apis/kmltext/..Kml-Google; (url), gzip - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pablo-almeida) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nallanikrithika) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: weps003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jasonlimcheewai) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: the-snyke) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sauravprakash-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: emranxaman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivankrisproxyserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~spowl-spowl) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~policevan2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: xg-prox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hide-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hackingclzpro) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyservr) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kevsproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~fox-eye) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~e-ddos) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: downarchivestuffproxyserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~amlloyd) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/33.0.1750.154 Safari/537.36 AppEngine-Google; (url; appid: s~ffffnm) zh-CN,zh;q=0.8,zh-TW;q=0.6 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp8) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 0.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
 0.facebook.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) zh-us,zh;q=0.9 https=1
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 -
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ASProxy/5.5b5 - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US, en;q=0.5 -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) en-US, en;q=0.5 -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0(compatible;GoogleBot/2.1;url) en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - https=1
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US, en;q=0.5 -
 www.google.com/bot.htmlimage/..GoogleBot/2.1 (url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlapplication/oggMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8,bn;q=0.6 -
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) . en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - https=1
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) . en-US, en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-GB -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) general.useragent.override en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compitable; GoogleBot/2.1; url) en-US, en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible;GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) general.useragent.override en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url ) en-US, en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) ru,en-us;q=0.7,en;q=0.7 -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.8 -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US https=1
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~ydachao-9) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.2 - -
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro006) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai04) en-US -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai10) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai15) en-US -
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) tr, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..(compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - -
 yandex.com/botstext/..198.143.47.1 Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - -
 yandex.com/botsimage/..77.67.27.22 Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url; libwww-perl/5.814; SYSTRAN) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) -
msn
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - https=1
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..(url) - -
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - -
crossref
 det.labs.crossref.orgapplication/jsonCrossRef DET 3.4.7 - url -
 det.labs.crossref.orgapplication/jsonCrossRef DET 3.4 - url -
 det.labs.crossref.orgapplication/jsonCrossRef DET 3.3.18 - url -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou inst spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..web spider/4.0(url) - -
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
commoncrawl
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* -
 zero.wikipedia.orgimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stuckkey url pt-BR,en,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.23.0 url - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/MusikAnimal url en-US,* -
 zero.wikipedia.orgtext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/wtmitchell url en-US,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stickee url en-AU,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Altsprachenfreund url de-DE,en,* -
 fr.wikipedia.org/wiki/Utilisateur:OrlodrimBotapplication/jsonOrlodrimBot/1.0 (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/I_dream_of_horses url en-US,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/KylieTastic url en-GB,* -
 ru.wikipedia.org/wiki/User:Karibekov_Vladislav_Y.application/jsonCiting Bot/1.1 (url; mail address ) - -
 en.wikipedia.orgtext/..wikidrain/1.0 (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Petrb url en-US,* -
 ko.wikipedia.orgapplication/jsonurl - -
 zero.wikipedia.orgtext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/9shi url zh-HK,en,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Jab843 url en-US,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Jogo.obb url de-DE,en,* -
quickiwiki
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/0.5.2 (url) PHP/5.3.13p1 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.19 - -
wikiwand
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.25.0-32-g0fd119d (url) en -
 scrapy.orgtext/..Scrapy/0.22.2 (url) en -
 scrapy.orgtext/..Scrapy/0.18.1 (url) en -
 scrapy.orgtext/..Scrapy/0.24.4 (url) en -
 scrapy.orgtext/..Scrapy/0.25.0-72-gd7d9f2f (url) en -
 scrapy.orgtext/..Scrapy/0.22.0 (url) en -
 scrapy.orgtext/..Scrapy/0.24.2 (url) en -
 scrapy.orgtext/..Scrapy/0.25.0-124-gc2497e6 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
cloudflare
 www.cloudflare.com/always-onlineimage/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
 www.cloudflare.com/always-onlinetext/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 https=1
zeerch
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - -
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - -
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - https=1
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - https=1
yacy
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.5.0-44-generic; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-32-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_11; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.41-042stab092.3; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.21-gentoo-r1; java 1.6.0_31; Japan/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_11; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.23.3.el6.x86_64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.20.5.el6.x86_64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmlimage/..yacybot (/global; amd64 Linux 3.13.0-32-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 FreeBSD 9.2-RELEASE-p9; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13-1-amd64; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab084.14; java 1.7.0_65; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.0-11-generic; java 1.8.0_20; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab092.2; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.2-gentoo; java 1.7.0_65; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; i386 Linux 3.2.0-4-686-pae; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.14.1-gentoo; java 1.7.0_65; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.16.0-11-generic; java 1.8.0_20; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.16.1-1-ARCH; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (allip-any; amd64 Linux 3.8.0-44-generic; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/pl) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab092.3; java 1.7.0_65; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p9; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.14.2-1-ARCH; java 1.7.0_65; UTC/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab078.28; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; x86 Windows XP 5.1; java 1.7.0_60; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Asia/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Etc/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab092.2; java 1.7.0_65; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - -
 tools.wmflabs.org/kmlexporttext/..url libwww-perl/6.03 - -
naver
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) - -
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 https=1
sblog
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) - -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; NLIL_IAHarvester url) - -
github
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/2.1.2p95 (url) en-us,en;q=0.5 -
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 wummel.github.io/linkchecker/text/..Mozilla/5.0 (compatible; LinkChecker/8.6; url) - -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/1.9.3p484 (url) en-us,en;q=0.5 -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
plos
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.3.5 - url - -
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.4 - url - -
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.3.8 - url - -
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.4.7 - url - -
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.4.8 - url - -
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.3.19 - url - -
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics 3.3.1 - url - -
 alm.plos.orgapplication/jsonArticle-Level Metrics 3.3.2 - url - -
 sfo-stage-alm01.int.plos.org/application/jsonPLOS Article-Level Metrics 3.2.2 - url - -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) - -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; image.coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
wordpress
 halokue.wordpress.comtext/..WordPress/4.0-beta2-20140728; url - -
 freedomfightersactualizetruths.wordpress.comtext/..WordPress/4.0-beta2-20140728; url - -
 hoangkimlong.wordpress.comtext/..WordPress/4.0-beta3-20140811; url - -
 goldendragon1971.wordpress.comtext/..WordPress/4.0-beta4; url - -
 hoangkimlong.wordpress.comtext/..WordPress/4.0-beta4; url - -
http:
 http://alm.example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - -
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url -
etao
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - -
crazywebcrawler
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.0, url - -
sputnik
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 -
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) fr-FR, fr;q=0.9, br-FR;q=0.7, gsw-FR;q=0.5, co-FR;q=0.3 -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) en-US, en;q=0.8, es-US;q=0.5 -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) it-IT, it;q=0.9 -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) en-GB, en;q=0.8, cy-GB;q=0.5 -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) es-ES, es;q=0.8, ca-ES;q=0.7, ca;q=0.6, eu-ES;q=0.5, eu;q=0.4, gl-ES;q=0.3, gl;q=0.2 -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) en-CA, fr-CA;q=0.9, en;q=0.8, fr;q=0.7, iu-Cans-CA;q=0.5, iu-Latn-CA;q=0.3, moh-CA;q=0.1 -
 cliqz.com/company/cliqzbotimage/..Cliqzbot/0.1 (url) de -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) vi-VN, vi;q=0.9 -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) tr-TR, tr;q=0.9 -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) hr-HR, hr;q=0.9 -
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) zh-SG, en-SG;q=0.8, zh;q=0.6, en;q=0.4 -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ru -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ar -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
wikilovesearth
 wikilovesearth.at/tools/text/..wle_at/1.0 (url; mail address ) - -
toolserver
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - -
example
 dev.example.orgapplication/jsonArticle-Level Metrics 3.3.1 - url - -
 www.example.com/text/..wikidrain/1.0 (url) - -
 example.orgapplication/jsonArticle-Level Metrics 3.3.14 - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.orgapplication/jsonArticle-Level Metrics 3.3-beta - url - -
 example.com/MyCoolTool/application/jsonMyCoolTool/1.1 (url; mail address ) BasedOnSuperLib/1.4 - -
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) - -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
 www.diffbot.comimage/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
riddler
 riddler.io/about.htmltext/..Riddler (url) - -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
xbmc
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1; url) - -
 xbmc.orgimage/..XBMC/13.2 Git:20140817-0f3db05 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130527-a36a92b (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.3;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Linux; Android; 3.0.36 armv7l; url) - -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.htmltext/..Feedly/1.0 (url; like FeedFetcher-Google) - -
wesee
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PageBot (url) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
 www.pinterest.com/text/..Pinterest/0.2 (url) - -
 www.pinterest.com/image/..Pinterest/0.2 (url) - -
114
 37.187.79.114/~lookformedical/yioop/bot.phptext/..Mozilla/5.0 (compatible; lookformedical2; url) -
 37.187.79.114/~lookformedical/yioop/bot.phptext/..Mozilla/5.0 (compatible; lookformedical2; url) - -
 37.187.79.114/~lookformedical/yioop/bot.phptext/..Mozilla/5.0 (compatible; lookformedical2; url) - https=1
 37.187.79.114/~lookformedical/yioop/bot.phpimage/..Mozilla/5.0 (compatible; lookformedical2; url) -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
aaronsw
 www.aaronsw.com/2002/feedfinder/text/..feedfinder/1.371 Python-urllib/1.16 url - -
findcan
 findcan.ca/bot.phptext/..Mozilla/5.0 (compatible; FindCanBot; url) - -
 findcan.ca/bot.phptext/..Mozilla/5.0 (compatible; Please Name Your robot; url) - -
 findcan.ca/bot.phptext/..Mozilla/5.0 (compatible; FindCanBot; url) - https=1
softvision
 softvision.com/application/jsonSvBot/1.0 (url; mail address ) - -
draigweb
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
 www.okian.ro/application/jsonMyBot/1.0 (url) - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
test
 test.co.kr/text/..xxxxxxxx/Nutch-2.2.1 (url) - -
edu
 pjwstk.edu.pltext/..Wiki Controversial/0.1; url ; mail address -
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - -
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..User-Agent: Drupal (url) - -
 drupal.org/text/..Drupal (url) -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) en-us,en;q=0.5 -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) - -
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
mytest
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
 duckduckgo.com/duckduckbot.htmlimage/..DuckDuckBot/1.1; (url) - -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
007ac9
 crawler.007ac9.net/text/..Mozilla/5.0 (compatible; 007ac9 Crawler; url) - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) -
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
uplabs0
 alm-1.uplabs0.comapplication/jsonArticle-Level Metrics 3.3.8 - url -
 alm-1.uplabs0.comapplication/jsonArticle-Level Metrics 3.3.7 - url -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
 www.saciol.com/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
wikidict
 www.wikidict.detext/..url - -
webthumbnail
 webthumbnail.orgimage/..Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; url) en,* -
 webthumbnail.orgtext/..Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; url) en,* -
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
verizon
 www.verizon.comapplication/jsonCrawler/1.0 (url; mail address ) Custom/1.0 - -
bigdatacorp
 ecommerce.bigdatacorp.com.br/faq.aspxtext/..Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; url) AppleWebKit/535.1 KHTML Chrome/13.0.782.107 Safari/535.1 pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4 -
site
 www.site.com/application/jsonBot/2.0 (url) -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) ja, en -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
newworldmodel
 newworldmodel.com/writing-jobtext/..WordPress/3.9.1; url - -
 newworldmodel.com/lotterytext/..WordPress/3.9.2; url - -
 newworldmodel.com/treatment-hemorrhoidstext/..WordPress/3.9.2; url - -
 newworldmodel.com/pure-colon-detoxtext/..WordPress/3.9.2; url - -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
bit
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) de
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) en
gorgor
 gorgor.irtext/..hivaBot/hivaBot-1.0 (Iranian Search Engine bot; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
go
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) ja -
 kc.nict.go.jp/project1/crawl.htmltext/..ICC-Crawler/2.0 (Mozilla-compatible; ; url) - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
grouphigh
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - -
100
 37.187.79.100/~lookformedical/yioop/bot.phptext/..Mozilla/5.0 (compatible; lookformedical; url) -
stad
 stad.comtext/..Mozilla/5.0 (compatible; stadbot/1.0; url) - -
SearchNearMe
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
perfectmarket
 www.perfectmarket.comtext/..Mozilla/5.0 (compatible; PerfectMarketKWTbot/1.0 url) - -
idmarch
 www.idmarch.org/bot.htmltext/..Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; url) en-us,en-gb,en;q=0.7,*;q=0.3 -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
simplepie
 simplepie.orgapplication/xmlSimplePie/1.3.1 (Feed Parser; url; Allow like Gecko) Build/20121030175403 - -
 simplepie.orgtext/..SimplePie/1.3.1 (Feed Parser; url; Allow like Gecko) Build/20121030175403 - -
livedoor
 reader.livedoor.com/application/xmllivedoor FeedFetcher/0.01 (url; 1 subscriber) - -
inagist
 inagist.comtext/..InAGist URL Resolver (url) en-us,en;q=0.5 -
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
geolocation
 www.geolocation.wsapplication/jsonGeolocation.ws import. url -
 www.geolocation.wsapplication/jsonGeolocation.ws Wikimedia Commons import. url -
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
metamagazine
 metamagazine.comtext/..WordPress/3.9.2; url -
 metamagazine.comtext/..WordPress/3.9.1; url -
mysite
 www.mysite.com/application/jsonMyBot/1.0 (url) - -
 www.mysite.com/text/..MyBot/1.0 (url) - -
 www.mysite.comtext/..Mozilla/5.0 (compatible; heritrix/${pom.version} url) - -
 www.mysite.com/botinfoapplication/vnd.php.serializedMozilla/5.0 (compatible; MyCoolBot/1.0; url) - -
 www.mysite.comimage/..Mozilla/5.0 (compatible; heritrix/${pom.version} url) - -
a6corp
 www.a6corp.com/a6-web-scraping-policy/text/..A6-Indexer/1.0 (url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
tineye
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
ac
 www.clips.ua.ac.be/patterntext/..Pattern/2.6 url - -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) - -
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.12 (url) - -
 tt-rss.org/application/xmlTiny Tiny RSS/1.13 (url) - -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) tr -
xovibot
 www.xovibot.net/text/..Mozilla/5.0 (compatible; XoviBot/2.0; url) - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
uptimetracker
 www.uptimetracker.com/text/..Mozilla/5.0 (compatible; uptimebot/2.0; url) - -
 www.uptimetracker.com/text/..Mozilla/5.0 (compatible; uptimebot/2.0; url) -
besaba
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) -
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
temnos
 temnos.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
servicesonrequest
 www.servicesonrequest.comtext/..User-Agent: WikiBot (url) - -
 www.servicesonrequest.com/application/jsonWikiBot(url) - -
twitter
 twitter.com/wiederfreitext/..wiederfreibot/1.0 (url) - -
QtWeb
testomato
 www.testomato.com/about/crawlertext/..minicrawler/2 url en-US,en,cs;q=0.01 -
crowsnest
 www.crowsnest.tv/text/..Crowsnest/0.5 (url) - -
creativepulses
 creativepulses.nltext/..CreativePulses Crawler (url) - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
wikimedia
 bits.wikimedia.orgimage/..- Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 -
 bits.wikimedia.orgtext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 -
meximas
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
mobdir
 mobdir.com/application/jsonMobdir/1.0 (url; mail address ) BasedOnSuperLib/1.4 -
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url) en-us -
fit4youpt
 fit4youpt.co.uk//imgtext/..WordPress/3.9.2; url - -
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
yougorhymes
 www.yougorhymes.com/site/rhyme-bottext/..RhymeBot/0.3 (url) - -
sprawk
 www.sprawk.com/text/..Sprawk/1.4 (url) - -
201846.419999976total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
pwb/rg3428 Pywikibot/2.0 - -
 application/json
 text/..
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - -
 application/vnd.php.serialized
 text/..
 application/json
GoogleBot-Image/1.0 - -
 image/..
 text/..
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
php wikibot classes - -
 application/vnd.php.serialized
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN -
 text/..
 image/..
 application/json
 application/xml
ClueBot/2.0 - -
 application/vnd.php.serialized
wikiwix-bot-3.0 - -
 text/..
LinkParser/2.0 - -
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
ProoXiBot fr -
 text/..
 application/ogg
py-radehh.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg3113 Pywikipediabot/2.0 - -
 application/json
maj_articles_recents/r3778 Pywikipediabot/2.0 - -
 application/json
 text/..
tigerbot - -
 application/json
 text/..
 image/..
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/json
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
pywikibot-compat-redirect.py/rg10993 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Bot/5.005006 - -
 application/json
PythonWikipediaBot/1.0 - -
 application/json
 text/..
 application/xml
Twitterbot/1.0 - -
 text/..
 image/..
 application/pdf
 application/ogg
 application/json
 video/webm
milog_bot/1.0 ( mail address ) - -
 text/..
DotNetWikiBot/2.99 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - -
 application/json
 text/..
DigitalsmithsBot - -
 text/..
pywikibot-1newar25.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
run_pagefromfile.3/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25(compatible;SogouMobileSpider/2.0) zh-cn -
 text/..
 -
UAA-UAA.py/r10417 Pywikipediabot/1.0 - -
 application/json
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 application/ogg
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - -
 application/json
 text/..
MediaWiki::Bot/3.2.6 - -
 application/json
SocialBot - -
 application/json
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
Owlin Bot v. 3.0 - -
 text/..
pywikibot-1newar25.py/rg11039 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-radeh.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
MediaWiki::Bot/3.005002 - -
 application/json
 text/..
ContextAd Bot 1.0 - -
 text/..
 image/..
 application/json
pywikipedia-imageuncat.py/rg10972 Pywikipediabot/1.0 - -
 application/json
 application/xml
Facebot/1.0 -
 text/..
 image/..
 application/xml
 application/ogg
maj_articles_recents/rg3649 Pywikibot/2.0 - -
 application/json
pywikibot-1newar25.py/rg11050 Pywikipediabot/1.0 - -
 application/json
pywikipedia-newinterwiki-fa.py/rg11028 Pywikipediabot/1.0 - -
 application/json
 application/xml
python-wikitools/1.2 (User:DickensBot) -
 application/json
 text/..
pywikibot-compat-commonscat.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - -
 application/json
 application/xml
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
WikidataBot framework - -
 application/json
RBot/0.3 ( mail address ) -
 application/json
 image/..
 text/..
spider - -
 text/..
 application/json
 application/vnd.php.serialized
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 - https=1
 text/..
pywikibot-1newar25.py/rg11035 Pywikipediabot/1.0 - -
 application/json
 application/xml
cwikt-interwiki.py/rg11033 Pywikipediabot/1.0 - -
 application/json
 application/xml
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - -
 text/..
compat-featured.py/r10441 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rs4639 Pywikibot/2.0 - -
 application/json
 text/..
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
anagrama/r-1 (unknown) Pywikibot/2.0 - -
 application/json
featured/s4911 Pywikibot/2.0 (User:) - -
 application/json
Google Bot - -
 application/json
DotNetWikiBot/3.0 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 (https://github.com/mwclient/mwclient) - -
 application/json
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
cat_guinness/-1 (unknown) Pywikibot/2.0 (User:RawBot) - -
 application/json
pywikibot-commonscat.py/r(unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
JackBot-fr.wikt.RemiseEnForme.py/r41 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
GermCrawler - -
 application/json
 text/..
 application/xml
User-Agent Baiduspider - -
 text/..
 image/..
pwb/r3673 Pywikibot/2.0 - -
 application/json
 text/..
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
MediaWiki::Bot/5.005007 - -
 application/json
YisouSpider - -
 text/..
 application/json
pywikipedia-zzfatranslator.py/rg10961 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
AnomieBOT/1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
 text/..
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
DeclensionParser/0.02 ( mail address ) - -
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - -
 text/..
 application/xml
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - -
 application/json
Screaming Frog SEO Spider/2,55 - -
 text/..
 image/..
 application/xml
compat-checkimages.py/rs11009 Pywikipediabot/1.0 - -
 application/json
CorenSearchBot/1.7 en libwww-perl/6.03 - -
 text/..
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - -
 application/json
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
KatBot bot.nytlabs.com> - -
 text/..
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
AnomieBOT/1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
 text/..
SecretaryBot (https://github/vlsergey/Secretary; mail address ) BasedOnJWBF - -
 text/..
 application/json
Webbot - -
 text/..
bot-headcat.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
BOT/0.1 (BOT for JCE) - -
 text/..
pywikipedia-git-maghalesaz2.py/r90 Pywikipediabot/1.0 - -
 application/json
compat-basictemplatedisamblist.py/rg11047 Pywikipediabot/1.0 - -
 application/json
HTMLParser/2.0 - -
 text/..
 image/..
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
pywikibot-compat-aaa_categorieAutori.py/rg10952 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-radeh.py/rg10961 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 -
 text/..
 -
pywikipedia-zzinter.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
bot-addhead.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-desc_bot.py/r90 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
BotReversor-BotReversor.py/rg11003 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.1.15 Darwin/14.0.0 en-us -
 image/..
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
featured/s4885 Pywikibot/2.0 (User:) - -
 application/json
compat-featured.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT/1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-1newar25.py/rg11034 Pywikipediabot/1.0 - -
 application/json
 application/xml
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - -
 application/json
interwiki/g3841 Pywikibot/2.0 (User:) - -
 application/json
Egerin Spider 1.0 - -
 text/..
PythonWikipediaBot/1.0 -
 application/json
 text/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
py-interwiki.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
Keywords/r-1 (unknown) Pywikibot/2.0 - -
 application/json
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 application/xml
Nutch Spider/Nutch-2.2.1 - -
 text/..
 image/..
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
ko_missing/-1 (unknown) Pywikibot/2.0 (User:Chobot) - -
 application/json
pywikipedia-zzfatranslator.py/rg11047 Pywikipediabot/1.0 - -
 application/json
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-importwdn3.py/r90 Pywikipediabot/1.0 - -
 application/json
run_replace.7r/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
py-navbox.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/rg11034 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/rg11050 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-git-del_new.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlBot) - -
 text/..
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
DotNetWikiBot/2.100 (Unix 3.2.0.60; ) - -
 text/..
 application/xml
XLinkBot/1.00 - -
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
maj_articles_recents/rg3630 Pywikibot/2.0 - -
 application/json
featured/s4853 Pywikibot/2.0 (User:) - -
 application/json
 text/..
SentiBot www.sentibot.eu (compatible with GoogleBot) - -
 text/..
featured/s4770 Pywikibot/2.0 (User:) - -
 application/json
German Wikipedia Broken Weblinks Bot; contact: mail address - -
 text/..
 image/..
pywikipedia-zzpol.py/rg11028 Pywikipediabot/1.0 - -
 application/json
mijnbots-NavSjabOpnemen.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
compat-redirect.py/rs11026 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
JackBot-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
GoogleBot-Image/1.0 zh-CN,zh;q=0.8,en;q=0.6 -
 image/..
 text/..
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
logpatrol/rg3680 Pywikibot/2.0 - -
 application/json
touchByCategory/g3814 Pywikibot/2.0 (User:CastorBot) - -
 application/json
Screaming Frog SEO Spider/2.55 - -
 text/..
 image/..
 application/ogg
pywiki-welcome.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
interwiki/g3839 Pywikibot/2.0 (User:ZuiarraBot) - -
 application/json
SurakWare MediaWiki Bot/1.0 - -
 text/..
 application/xml
SMcrawler - -
 text/..
 image/..
compat-featured.py/rg11050 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg11033 Pywikipediabot/1.0 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
articles/r2554 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-aaa_categorieTesti.py/rg10952 Pywikipediabot/1.0 - -
 application/json
 application/xml
MFU Spider(167)/Nutch-2.2.1 - -
 text/..
pywikibot-compat-commonscat.py/rg11039 Pywikipediabot/1.0 - -
 application/json
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikilab-bot_viquirepte.py/rg11003 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/r10843 Pywikipediabot/1.0 - -
 application/json
compat-redirect.py/rs10992 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - -
 application/json
www.integromedb.org/Crawler - -
 text/..
 application/json
 image/..
pywikipedia-git-wdlabel.py/rg90 Pywikipediabot/1.0 - -
 application/json
redirect/s4911 Pywikibot/2.0 (User:Xqbot) - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
featured/s4845 Pywikibot/2.0 (User:) - -
 application/json
pywikipedia-git-importwdn2.py/r90 Pywikipediabot/1.0 - -
 application/json
360spider-image - -
 image/..
t9/rg2816 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
compat-checkimages.py/rs11026 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-checkimages.py/rg10976 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X; en-us) AppleWebKit/534 KHTML Version/7.0 Mobile/11A465 Safari/9537.53 MsnBot-Media /1.0b - -
 text/..
 image/..
 -
rijksmonumenten_import/g3822 Pywikibot/2.0 (User:BotMultichill) - -
 application/json
PHPCrawl - -
 text/..
pywikibot-1newar25.py/rg11037 Pywikipediabot/1.0 - -
 application/json
pwb/rg3012 Pywikipediabot/2.0 - -
 application/json
pywikipedia-substub.py/r11028 Pywikipediabot/1.0 - -
 application/json
Rukhabot/0.1 (https://en.wiktionary.org/wiki/User:Rukhabot) - -
 application/json
pwb/r3515 Pywikibot/2.0 - -
 application/json
compat-basicarticledisamblist.py/rg11050 Pywikipediabot/1.0 - -
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
pywikipedia-zzinter.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 image/..
 application/ogg
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
Livelapbot/0.1 en-US,en -
 text/..
 image/..
 application/pdf
RSS Bot/1.8 (Mac OS X Version 10.9.4 (Build 13E28)) en;q=1, ja;q=0.9, fr;q=0.8, de;q=0.7, es;q=0.6, it;q=0.5 -
 application/xml
 -
JavaCrawler/1.1 - -
 text/..
InfolinksSpider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
 application/json
web-crawler en-US,en;q=0.8 -
 text/..
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
porzucone/g3820 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
AnomieBOT/1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
RokaSpider/Nutch-1.8 (' mail address ') de-de,de,en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; en-US) AppleWebKit/537.36 KHTML Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler en-US,* -
 text/..
 application/json
GoogleBot-Image/1.0 bg -
 text/..
 image/..
pywikibot-compat-commonscat.py/rg11037 Pywikipediabot/1.0 - -
 application/json
 application/pdf
 application/xml
COIBot/1.00 - -
 text/..
featured/s4757 Pywikibot/2.0 (User:) - -
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - -
 text/..
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
porzucone/g3789 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
pywikipedia-bot_rss.py/rg14302 Pywikipediabot/1.0 - -
 application/json
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
commonscats/r2554 Pywikipediabot/2.0 - -
 application/json
maj_articles_recents/rg3608 Pywikibot/2.0 - -
 application/json
pywikibot-compat-sichteroptin.py/r10883 Pywikipediabot/1.0 - -
 text/..
 application/json
exintro_bot/0.1 - -
 application/json
pywikipedia-git-importwdn3.py/rg90 Pywikipediabot/1.0 - -
 application/json
ArchiveTeam ArchiveBot/20140810.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/36.0.1985.125 Safari/537.36 - -
 image/..
 text/..
 application/ogg
DotNetWikiBot/2.96 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
pywikipedia-newar.py/rg11050 Pywikipediabot/1.0 - -
 application/json
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
ArchiveTeam ArchiveBot/20140804.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/36.0.1985.125 Safari/537.36 - -
 image/..
 text/..
2971799/g3742 Pywikibot/2.0 (User:RussBot) - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
py-interwiki.py/rs11026 Pywikipediabot/1.0 - -
 application/json
 application/xml
ko_missing2/r-1 (unknown) Pywikibot/2.0 - -
 application/json
3358130/g3873 Pywikibot/2.0 (User:RussBot) - -
 application/json
cd-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
featured/s4837 Pywikibot/2.0 (User:) - -
 application/json
pywikipedia-git-featured_new.py/r90 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; SemanticBot/1.1) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
pwb/rg3536 Pywikibot/2.0 - -
 application/json
pywikibot-compat-pwb.py/rg10940 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pywiki-cfd.py/rg11032 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - -
 text/..
censo4b/-1 (unknown) Pywikibot/2.0 (User:KLBot2) - -
 application/json
 text/..
featured/s4820 Pywikibot/2.0 (User:) - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 text/..
 -
 application/json
Niki-Bot - -
 text/..
Pastec bot -
 image/..
 text/..
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
Inlibris.com XMLBot/1.0 - -
 text/..
compat-featured.py/rg11034 Pywikipediabot/1.0 - -
 application/json
featured/s4814 Pywikibot/2.0 (User:) - -
 application/json
NeweraHPC BOT Mozilla/5.0 - -
 text/..
GoogleBot - -
 text/..
compat-featured.py/rg11005 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-git-importwdn.py/r90 Pywikipediabot/1.0 - -
 application/json
AnomieBOT/1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-_move2wikidata_taxon.py/rg10961 Pywikipediabot/1.0 - -
 application/json
anagrama/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
pywikipedia-git-maghalesaz_new.py/r90 Pywikipediabot/1.0 - -
 application/json
JackBot-fr.wikt.RemiseEnForme.py/r44 Pywikipediabot/1.0 - -
 application/json
pywikilab-llista_mils.py/rg11003 Pywikipediabot/1.0 - -
 application/json
node.js - nodemw - mail address - Maintainer: Rainer Rillke - mail address - [[:commons:User:Rillke]] - -
 application/json
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
AnomieBOT/1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
compat-featured.py/rg11037 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/3.10 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
pwb/rs4638 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 -
 text/..
 application/json
Crawler/1.2 - -
 text/..
pywikipedia-zzaddnavebox.py/rg11047 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/rg10957 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzpurging.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; GoogleBot/2.1; en-US,en;q=0.5 -
 text/..
 image/..
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - -
 text/..
 application/json
 application/xml
pywikipedia3-welcome.py/r11747 Pywikipediabot/1.0 - -
 application/json
Python27-pythonw.exe/r(unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (compatible; MerchantCentricBot/1.0) en-US,* -
 text/..
compat-desc_bot_nn.py/rg11047 Pywikipediabot/1.0 - -
 application/json
porzucone/g3741 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
robot leapit.com en-US,* -
 image/..
 text/..
porzucone/g3779 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-importwdq6.py/r90 Pywikipediabot/1.0 - -
 application/json
wiki_crawler/r-1 (unknown) Pywikibot/2.0 - -
 application/json
porzucone/g3793 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
wymowa/g3810 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
compat-sports-reference.py/rg11050 Pywikipediabot/1.0 - -
 application/json
redirect/s4820 Pywikibot/2.0 (User:Xqbot) - -
 application/json
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
php wikibot classes -
 application/vnd.php.serialized
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - -
 application/xml
 text/..
Baiduspider - -
 image/..
 text/..
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-desc_bot_da.py/rg11047 Pywikipediabot/1.0 - -
 application/json
Projet Sport-Projetsportliste2.py/r(unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/g3907 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
Update_items_commonscat_communes/r2554 Pywikipediabot/2.0 - -
 application/json
wymowa/g3789 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
wymowa/g3817 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
wymowa/g3798 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
wymowa/g3862 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
iw_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; SputnikImageBot/2.2) - -
 image/..
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
 -
 text/..
porzucone/g3894 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
My Bot - -
 text/..
citazioni_bot/rg3343 Pywikibot/2.0 - -
 application/json
compat-desc_bot_sv.py/rg11047 Pywikipediabot/1.0 - -
 application/json
redirect/s4885 Pywikibot/2.0 (User:Xqbot) - -
 application/json
MoodleBot/1.0 - -
 application/xml
 text/..
 application/vnd.php.serialized
 image/..
porzucone/rg3621 Pywikibot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg11035 Pywikipediabot/1.0 - -
 application/json
 application/xml
redirect/s4770 Pywikibot/2.0 (User:Xqbot) - -
 application/json
porzucone/g3884 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
archiv-erl-bot/rg2953 Pywikipediabot/2.0 - -
 application/json
porzucone/g3828 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
 text/..
compat-welcome.py/rg11037 Pywikipediabot/1.0 - -
 application/json
porzucone/g3910 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
porzucone/g3862 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
dmashine-CleanupTematic.py/r10391 Pywikipediabot/1.0 - -
 application/json
wymowa/g3793 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
EarwigBot/0.2.dev.gitd411f55c (Python/2.7.3; https://github.com/earwig/earwigbot; mail address ) - -
 application/json
porzucone/g3880 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
porzucone/g3913 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
porzucone/g3856 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
porzucone/g3753 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
porzucone/g3841 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
porzucone/rg3678 Pywikibot/2.0 - -
 application/json
compat-desc_bot.py/rg11047 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/rg11033 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzgallery.py/rg11047 Pywikipediabot/1.0 - -
 application/json
 application/xml
vm-auto-erl-neu/s4885 Pywikibot/2.0 (User:Xqbot) - -
 application/json
porzucone/g3891 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; fr-fr; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 fr-FR, en-US -
 image/..
 -
 text/..
 application/json
porzucone/rg3649 Pywikibot/2.0 - -
 application/json
wikibot-wb-welcome.py/rg10972 Pywikipediabot/1.0 - -
 application/json
svn-compat-commonscat.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
porzucone/g3692 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
wymowa/g3837 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
pywikipedia-featured.py/r(unknown) Pywikipediabot/1.0 - -
 application/json
porzucone/g3871 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
pwb/r3152 Pywikipediabot/2.0 - -
 application/json
 text/..
pywikipedia-zzfawikipatroler.py/rg11047 Pywikipediabot/1.0 - -
 application/json
porzucone/g3904 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
interwiki/g3841 Pywikibot/2.0 (User:Whymbot) - -
 application/json
py-importwdn.py/rs11026 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7600.0; ) - -
 text/..
 application/xml
WikiBot/0.1 - -
 text/..
porzucone/g3814 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
porzucone/g3777 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
porzucone/rg3637 Pywikibot/2.0 - -
 application/json
pywikipedia-commonscat.py/rg11050 Pywikipediabot/1.0 - -
 application/json
wymowa/g3704 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
pywikipedia-zzredirectyeh.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
cdh-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
Screaming Frog SEO Spider/2.40 - -
 text/..
 image/..
porzucone/g3802 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
redirect/s4853 Pywikibot/2.0 (User:Xqbot) - -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 0.5 - -
 application/json
wymowa/g3820 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 text/..
 -
porzucone/g3724 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
python-wikitools/1.2 (User:Mr.Z-bot) - -
 application/json
wymowa/g3845 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
DotNetWikiBot/2.101 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
 application/xml
porzucone/g3902 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
gsa-crawler (Enterprise; T3-PWEV36BT9WWGY; mail address ) - -
 text/..
pywikipedia-newar.py/rg11037 Pywikipediabot/1.0 - -
 application/json
 application/xml
erfgoedbot-categorize_images.py/rg10979 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/g3871 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
Phantom.js bot cs-CZ,en,* -
 image/..
 text/..
pywikipedia-zzredirecthe.py/rg11028 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-featured.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
 application/json
GoogleBot-Image/1.0 -
 text/..
 image/..
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - -
 application/json
wymowa/g3823 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
AnomieBOT/1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]]) - -
 application/json
wymowa/g3904 Pywikibot/2.0 (User:Alkamid) - -
 application/json
ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/36.0.1985.125 Safari/537.36 - -
 image/..
 text/..
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) - -
 text/..
pywiki-category.py/rg11032 Pywikipediabot/1.0 - -
 application/json
endeca webcrawler/WebCrawler en-us, en-gb, en;q=0.7, *;q=0.3 -
 text/..
frettiebot-python.py/rg11052 Pywikipediabot/1.0 - -
 application/json
-c/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
wymowa/rg3645 Pywikibot/2.0 - -
 application/json
Botzilla-av_bot.py/rg11003 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.1.14 Darwin/14.0.0 en-us -
 image/..
 text/..
pywikipedia-git-new_iw14.py/r90 Pywikipediabot/1.0 - -
 application/json
update-regular-users/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
wymowa/g3685 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
pywikipedia-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
GoogleBot (gocrawl v0.4) - -
 text/..
getMissingVitalArticles/r3152 Pywikipediabot/2.0 - -
 application/json
CaBot - -
 text/..
wymowa/g3902 Pywikibot/2.0 (User:Alkamid) - -
 application/json
rijksmonumenten_import/rg3590 Pywikibot/2.0 - -
 application/json
pywikipedia-lit.py/rg11050 Pywikipediabot/1.0 - -
 application/json
CaBot Script (running on nightshade.toolserver.org) - -
 application/vnd.php.serialized
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
Mozilla/5.0 (compatible; MojeekBot/0.6; https://www.mojeek.com/bot.html) - -
 text/..
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1; es, en, *
 text/..
 image/..
ArchiveTeam ArchiveBot/20140801.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/36.0.1985.125 Safari/537.36 - -
 image/..
 text/..
wymowa/g3734 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
wymowa/g3873 Pywikibot/2.0 (User:Alkamid) - -
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1 - -
 image/..
 text/..
pywikipedia-interwiki.py/rg8 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/g3777 Pywikibot/2.0 (User:AlkamidBot) - -
 application/json
wymowa/g3914 Pywikibot/2.0 (User:Alkamid) - -
 application/json
35539.3000000001total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Sep 29, 2014 22:49
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers