Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Jul 2014 - 31 Jul 2014
000 ⇒ k
 
 This analysis is based on a 1:1000 sampled server log (squids)

 Other reports:
 Requests: Destination/Mime - Origin - Methods - Scripts - User agents - Skins - Crawlers - Op.Sys. - Browsers - Google - Country data

 Notes on reliability of these data

 Unresolved Bugzilla bugs: 46190, 46191, 46195, 46201, 46265, (46267), 46268, 46269, 46271, 46273, 46274, 46275, 46277, 46278, 46279, 57376

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 147,852,810 page requests (mime type text/html only!) per day are considered crawler requests, out of 626,563,000 external requests, which is 23.6%

Page requests for crawlers that specify a url in the agent string
Count
x 1000		Secondary domain
(~site) name		URLMime typeUser agent
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) - https=1
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) - -
 www.bing.com/bingbot.htmtext/..10.23.42.20 Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) - -
 www.bing.com/bingbot.htmapplication/jsonMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url) -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) WikipediaAPI - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf301-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5
 www.bing.com/bingbot.htmtext/..54.215.150.37 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~yourrevenues-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6009-hrd) - -
 www.bing.com/bingbot.htmtext/..54.241.179.159 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython6-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~swedishdanish-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf603-hrd) - -
 www.bing.com/bingbot.htmtext/..54.215.150.33 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf712-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~scroll-down-hrd) - -
 www.bing.com/bingbot.htmtext/..54.215.148.134 Mozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk-hrd) - -
 www.bing.com/bingbot.htmapplication/xmlMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6000-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxycgi1-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk9-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~they-encounter-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython7-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile8-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~tutorialses-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surfproxy4-hrd) - -
 www.bing.com/bingbot.htmapplication/oggMozilla/5.0 (compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxypython9-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai07) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6001-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxydisk5-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf702-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy6002-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxy5070-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~goodersearch-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf206-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~guidesites-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~response-out-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~usethisproxy-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro006) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro007) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~proxyfile2-hrd) - -
 www.bing.com/bingbot.htmimage/..(compatible; bingbot/2.0; url) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro005) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pyproxy4-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai04) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~surf605-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~ddbrite-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pyproxy0-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~i-am-proxy-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~appcodes-hrd) - -
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) en-us,en;q=0.5 -
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/p/crawler4j/text/..crawler4j (url) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 docs.google.comtext/..Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; url) - -
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freebase-refinery) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~skawa-easyling) - -
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew) - -
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alexliao1995) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9329269) - -
 script.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleApps script; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kampungbebas) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaxminds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webslinger81) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thearakanesemeetingpoint) - -
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mad-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~vulture-engine) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: my-reg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 42turkeysproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ax4413) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~adddon1) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sastraproxy) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: andrewexploit) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 9000tunnels) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freesurf003) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ideserveinternet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kikopea-openproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imggrabberredirect) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jg96graham) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mosesproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer) - -
 code.google.com/appenginetext/..Python-urllib/2.5 AppEngine-Google; (url; appid: s~isnt-it) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dadolphson-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~neidlingermj1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openfence) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ambeaujeanc) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: darecountyschools) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chiprut-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: quigonjinn04) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nyinayminproxy9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy-goo) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-ba-k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~s-alexander) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~silixd-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hamgerbur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: cravibruce) - -
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: discretepword) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server) - -
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: martin.heppATunibw.de) AppEngine-Google; (url; appid: gr4bing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: godfatherabhi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ghost-surf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: icanjango) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jackieonthefloor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: freetobrowse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: predictionwrong) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sunmintgreen2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: alex2610ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ownsterserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy12345) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sjbrundage123456789) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dex-dwds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tgbeeson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zmomyanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go-online-now) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~schoolruiner) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wmhsonline) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rxrwng) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bassgnt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ridemyhell) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sony-hack) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bubba-ps) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bel3afya) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: student-pupil-help) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: taterproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: riccio-hacks-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vi-mobile) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: findmory) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: slobozincur) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thegatetoweb) - -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thepeterdeutsch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: elbigboss-35) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tortelliniman) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: posten604) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jakestokes) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: drrkproxxxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tunisistan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~sl-santabanta) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nashma4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 100pui) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: carlos-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ethupbolt) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nhsportal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: computer-solutions) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mygale1975) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~clon-games) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~rennnat) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: velocity-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webduplicate) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kha1989led) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: keiths-proxy-server) - -
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.2; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0 AppEngine-Google; (url; appid: s~ad0066e) zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dexapassa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usingomega) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jerproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mahshighschool) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: img-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tentativiinutili) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~misterhac) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mogoktharproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: jasonlimcheewai) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~samson-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: minecraft6699) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline0) - -
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~kyouen3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mbw-portal) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ihack-tunnel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wiwohk-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: internet-comunidadmoviles) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gregatr0n) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~no8lancelot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq3) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kasumiremix) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: islom-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kakuro69) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: khsprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~projection11111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: techjawaproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: go6internet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: imabouncytigger) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~net-people) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: clh-sw) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hideproxyz) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: m19631005) - -
 code.google.com/appengine-AppEngine-Google; (url; appid: vi-mobile) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: time-ghost) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: leemus-net-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: the-snyke) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~secretinternet2011-2012) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yuricamara) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~hide-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~pridvidhar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlq5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rabbit-hole-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~jaydenpwns1) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nation4india) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thinkersearch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-data) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: sonydonin) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~win8wale) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kerouanen) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: xg-prox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitaniel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: xbmacx) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~policevan2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mjatsoft-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azzaziprxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: its-a-tr) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~pey3be) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~inter-land) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: scraperproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: shoby-proxy-server) - -
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tek-app) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~freekaznet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: longbows-hideout) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kaveriselvaraj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~getwebaddress) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~skyrideraj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~simolifet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~techdatproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mistakeproxyarea) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: raja584sekhar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~e-ddos) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: irfansurf) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: unblockwifi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: this-is-not-what-u-think) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~oegishelm) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: electrofitomms) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kevsproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy0) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lunaslunacy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thanhtut88) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lil05warrior) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: red-arg) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pleasesendmeto) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaucho-labnol) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~italiatiki) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~badbolt20) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: klobotdoor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: izzy-queenofthefae) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abhorsen009) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: unlimitedbrowsing) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaxminds) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~private-eye-ear) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 1free-surfing) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: demostene) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~only-bits) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 12345proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline7) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: puthiyathiravidan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ritroxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kittehserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~fox-eye) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hackingclzpro) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: argim-free) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: digitalhunch-freenet) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-phpproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vietwebdisk) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tofi2k) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~aman11dh) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebproxy8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: batamsearch) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: madkaiser2) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: beansacks) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~ttpllax) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kushgenius) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: lovelamp1988) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crowdsurfer100) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~toxic-panda) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vadim-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy4free2me) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: iprithi) - -
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~psjimi) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: capitan300) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~secure-facebook-ranney) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: aboytes13tls) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~certificate-check-403) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mboharsik) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: zabastan) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-proxy-hh) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxitalia) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: yblk-ext-proxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wanttwitter) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~kyaysarlay) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: randohnson) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~alekproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~twofrostyfoyou) - -
 www.google.com/bot.htmlapplication/pdfMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: korvas-sux) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~lpuums29) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: myproxywx) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivankrisproxyserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp6) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxy3plus) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ucsm111) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kylesumrallproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: itsmetherobot) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: free-myanmar) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davidgotmoney50) - -
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~munnaproxyapp8) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nashimlive-nashimnx) - -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web-mastered) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mrjosiaht) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bakapouetprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dustbunnytycoonmonitor) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-4) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mitchblows) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ikaryse) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gaproxyserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~warawasetekure) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: funnyinternetwebsite) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dr-use-me) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~shoutscreamprxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kuryproxy) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~batshiitinsane) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: gratistelcel) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: itechgiz-proxy-server) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: trabserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~lg-mg230) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tacomansterprox) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hoptheborder) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~munnaproxyapp17) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy202) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: caineserver) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nuraiitj) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kappasan-mirror) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl5) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: encalada-proxy-one) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~amort141) - -
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~nishanthreddysingham) - -
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url) - -
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url) -
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url) - -
 0.facebook.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 0.facebook.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.facebook.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 developers.facebook.comimage/..facebookplatform/1.0 (url) -
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) -
 www.baidu.com/search/spider.htmimage/..Baiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spi_der.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-us,en;q=0.5 -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - https=1
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.html-Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider-cpro; url) - -
 www.baidu.com/search/spider.htmltext/..(compatible; Baiduspider/2.0; url) en-US
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~pagedaili7) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US https=1
 www.baidu.com/search/spider.htmapplication/xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/rsd+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmapplication/opensearchdescription+xmlBaiduspider-image(url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~ydachao-9) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai04) en-US -
 www.baidu.com/search/spider.htmapplication/jsonBaiduspider-image(url)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~zhdai15) en-US -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) - -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: s~mengpro006) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (Linux;u;Android/2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; url) zh-cn,zh-tw -
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) en-US -
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) ASProxy/5.2 - -
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) - -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-CN,zh;q=0.8,en;q=0.6 -
 www.google.com/bot.htmltext/..(compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - https=1
 www.google.com/bot.html-(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url) -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) . en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible;GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url). en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-us -
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible;GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) bg -
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - https=1
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - https=1
 www.google.com/bot.htmlimage/..(compatible; GoogleBot/2.1; url) - -
 www.google.com/text/..mozilla/5.0(compatible; GoogleBot/2.1 url bot.html) en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..Mozilla/5.0(compatible: GoogleBot/2.1;url) en-US,en;q=0.5 -
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) zh-cn -
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url) - -
 www.google.com/bot.htmltext/..compatible; GoogleBot/2.1; url en-US,en;q=0.5 -
 www.google.com/bot.htmltext/..(iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..- Mozilla/5.0 (compatible; GoogleBot/2.1; url) en-US,en;q=0.5 -
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; GoogleBot/2.1; url) - -
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; ) - -
yahoo
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRJ/YATS crawler (url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))' - -
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) - https=1
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url) en,* -
 help.yahoo.com/help/us/ysearch/slurpapplication/oggMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp; url) - -
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..Y!J-BRU/VSIDX dead link checker (url) - -
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..(compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url) -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) tr, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexBot/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 https=1
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botstext/..(compatible; YandexImages/3.0; url) - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) en-us, en;q=0.7, *;q=0.01 -
 yandex.com/botstext/..10.23.42.20 Mozilla/5.0 (compatible; YandexBot/3.0; url) (via Web-Blaster/2.21 (http://www.a-blast.org/web-blast.html)) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) de, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/oggMozilla/5.0 (compatible; YandexBot/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01 -
 yandex.com/botsapplication/xmlMozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url) ru, uk;q=0.8, be;q=0.8, en;q=0.7, *;q=0.01
 yandex.com/botsapplication/jsonMozilla/5.0 (compatible; YandexImages/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url) - -
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; url)0 readers - -
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexAntivirus/2.0; url) ru-RU
 yandex.com/botsimage/..(compatible; YandexBot/3.0; url) - -
digplanet
 www.digplanet.com/wikiapplication/vnd.php.serializedDigplanet/1.0 (url; mail address ) PHP/5.4 - -
msn
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url) - https=1
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-media/2.0b (url) - -
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url) - -
 search.msn.com/msnbot.htmtext/..(url) - -
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url) - -
cibra
 cibra.de/text/..CiBra Data Collector (url) - -
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/image/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/application/jsonMozilla/5.0 (compatible; AhrefsBot/5.0; url) - -
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/5.0; url) -
 ahrefs.com/robot/text/..(compatible; AhrefsBot/5.0; url) - -
cliqz
 cliqz.com/company/cliqzbottext/..Cliqzbot/0.1 (url) de -
 cliqz.com/company/cliqzbotimage/..Cliqzbot/0.1 (url) de -
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url) zh-cn -
 www.sogou.com/docs/help/webmasters.htm#07application/jsonSogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url) - -
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou inst spider/4.0(url) zh-cn -
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) - -
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url) en,*
mail
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotsimage/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; url) - -
 go.mail.ru/help/robotstext/..Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1 -
 go.mail.ru/help/robotstext/..(compatible; Linux x86_64; Mail.RU_Bot/2.0; url) ru,by;q=0.7,en;q=0.7,kz;q=0.7,ua;q=0.7,*;q=0.1
orange
 wikipedia.orange.fr/text/..API/1.0 (url) - -
 wikipedia.orange.fr/application/xmlAPI/1.0 (url) - -
commoncrawl
 commoncrawl.org/faq/text/..CCBot/2.0 (url) en-us,en-gb,en;q=0.7,*;q=0.3 -
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
 the.echonest.com/reader/image/..nestReader/0.3 (discovery; url; reader at echonest.com) en -
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) - -
 www.text/..Mozilla/5.0 (compatible; GoogleBot/2.1 urlGoogleBot.com/bot.html) - -
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html) -
 www.text/..Mozilla/5.0 (compatible; (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 KHTML Chrome/29.0.1547.62 Safari/537.36/3.1.2-SNAPSHOT-20130905-1227 urlGoogleBot.com) - -
wikipedia
 de.wikipedia.org/wiki/Benutzer:APPER/WikiHistorytext/..WikiHistory (url) - -
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Frosty url en-AU,* -
 zero.wikipedia.orgimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.23.0 url - -
 zero.wikipedia.orgtext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-US,en;q=0.5 -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/wtmitchell url en-US,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Solarra url en-US,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/RationalBlasphemist url en-CA,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Altsprachenfreund url de-DE,en,* -
 zero.wikipedia.orgimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) en-us,en;q=0.5 -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/MusikAnimal url en-US,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/The_Last_Arietta url en-US,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Natsu83 url de-DE,en,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Petrb url en-US,* -
 zero.wikipedia.orgtext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url @www.google.com/bot.html) - -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/KylieTastic url en-GB,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/yxyang93 url zh-CN,en,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Stuckkey url pt-BR,en,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Josve05a url sv-SE,en,* -
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/Spuk968 url de-DE,en,* -
quickiwiki
 www.quickiwiki.comtext/..QuickiWiki/1.0 (url; mail address ) BasedOnNodeJs/0.10.23 - -
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url) - -
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.12.21-gentoo-r1; java 1.6.0_31; Japan/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.13-gentoo; java 1.7.0_55; GMT09:00/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_11; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-32-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; i386 Linux 3.2.0-4-686-pae; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (-global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab090.5; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.20.3.el6.x86_64; java 1.7.0_65; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; x86 Windows 7 6.1; java 1.7.0_51; Asia/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_31; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_60; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 FreeBSD 9.2-RELEASE-p9; java 1.7.0_60; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_11; Europe/ru) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-30-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.11.0-24-generic; java 1.7.0_55; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; arm Linux 3.8.13.23; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-29-generic; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_21; America/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 2.6.32-042stab078.28; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.11-2-amd64; java 1.7.0_21; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.11.10-17-desktop; java 1.7.0_55; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.21-gentoo-r1; java 1.6.0_31; Japan/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_51; Asia/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/fr) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.12.21-gentoo-r1; java 1.6.0_31; Asia/ja) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.15.6-1-ARCH; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; x86 Windows XP 5.1; java 1.7.0_60; Europe/de) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (/global; i386 Linux 3.2.0-4-686-pae; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-431.20.5.el6.x86_64; java 1.7.0_65; Europe/en) url en-us,en;q=0.5 -
scrapy
 scrapy.orgtext/..Scrapy/0.20.1 (url) en -
 scrapy.orgtext/..Scrapy/0.25.0-32-g0fd119d (url) en -
 scrapy.orgtext/..Scrapy/0.22.2 (url) en -
 scrapy.orgtext/..Scrapy/0.23.0-261-g7fd8a1a (url) en -
 scrapy.orgtext/..Scrapy/0.24.2 (url) en -
 scrapy.orgtext/..Scrapy/0.22.0 (url) en -
 scrapy.orgtext/..Scrapy/0.16.5 (url) en -
 scrapy.orgtext/..Scrapy/0.18.1 (url) en -
 scrapy.orgtext/..Scrapy/0.25.0-72-gd7d9f2f (url) en -
 scrapy.orgimage/..Scrapy/0.22.0 (url) en -
naver
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 -
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ko,en;q=0.5 https=1
 help.naver.com/robots/text/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/image/..Yeti/1.1 (Naver Corp.; url) ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4 -
 help.naver.com/robots/application/jsonYeti/1.1 (Naver Corp.; url) - -
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url ) - -
 pear.php.net/image/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.1.1 (url) PHP/5.3.15 - -
 pear.php.net/text/..PEAR HTTP_Request class ( url ) - -
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.18 - -
 pear.php.net/package/http_request2text/..HTTP_Request2/2.2.0 (url) PHP/5.3.28 - -
 pear.php.net/package/http_request2image/..HTTP_Request2/0.5.2 (url) PHP/5.3.13p1 - -
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url) - -
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); url) - -
sblog
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2; url) cs -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test1; url) cs -
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; url) cs,cz,sk;q=0.7,*;q=0.5 -
 fulltext.sblog.cz/text/..Mozilla/5.0 (compatible; SeznamBot/3.2-test2; url) * -
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url) cs,cz,sk;q=0.7,*;q=0.5 -
wmflabs
 tools.wmflabs.org/geohacktext/..Geohack (url) - -
 tools.wmflabs.org/languagetool/application/xmlurl bot, contact: mail address - -
 tools.wmflabs.org/kmlexporttext/..url libwww-perl/6.03 - -
plos
 alm.plos.orgapplication/jsonPLOS Article-Level Metrics 3.1.1 - url - -
 sfo-stage-alm01.int.plos.org/application/jsonPLOS Article-Level Metrics 3.1.1 - url - -
 sfo-stage-alm01.int.plos.org/application/jsonPLOS Article-Level Metrics 3.2.2 - url - -
findcan
 findcan.ca/bot.phptext/..Mozilla/5.0 (compatible; FindCanBot; url) - -
 findcan.ca/bot.phptext/..Mozilla/5.0 (compatible; FindCanBot; url) - https=1
 findcan.ca/bot.phpimage/..Mozilla/5.0 (compatible; FindCanBot; url) - -
easou
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlapplication/jsonMozilla/5.0 (compatible; EasouSpider; url) - -
 www.easou.com/search/spider.htmlapplication/oggMozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; EasouSpider; url) zh;q=0.9,en;q=0.8 -
 www.easou.com/search/spider.htmltext/..Mozilla/5.0 (compatible; EasouSpider; url) - -
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
 www.archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; special_archiver/3.1.1 url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; url) - -
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140628-0002 url) - -
 archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url) - -
pagefreezer
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/image/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 -
 pagefreezer.com/pagefreezer-crawler/text/..PageFreezer (pagefreezer crawler; url; mail address ) en-us,en;q=0.5 https=1
etao
 open.etao.com/dev/EtaoSpidertext/..Mozilla/5.0 (compatible; EtaoSpider/1.0; url) - -
sputnik
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 -
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) ru,en;q=0.1,*;q=0.01 https=1
 corp.sputnik.ru/webmastertext/..Mozilla/5.0 (compatible; SputnikBot/2.3; url) - -
pinterest
 pinterest.com/text/..Pinterest/0.1 url - -
 pinterest.com/image/..Pinterest/0.1 url - -
wikiwand
 www.wikiwand.comapplication/jsonWikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
 www.wikiwand.comtext/..WikiWand/0.1 (url; mail address ) BasedOnNodeJs/0.10.23 - -
coccoc
 help.coccoc.com/text/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
 help.coccoc.com/image/..Mozilla/5.0 (compatible; coccoc/1.0; url) en-us;q=0.7,en;q=0.3 -
zeerch
 www.zeerch.com/zeerch2/bot.phptext/..Mozilla/5.0 (compatible; LA2; url) - -
 www.zeerch.com/bot.phptext/..Mozilla/5.0 (compatible; LA1; url) - -
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url) - -
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url) - -
yapq
 yapq.comapplication/jsonYapq mail address url - -
cloudflare
 www.cloudflare.com/always-onlineimage/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en-US,* -
 www.cloudflare.com/always-onlineimage/..Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; url) AppleWebKit/534.34 en,* -
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url) - -
stackoverflow
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url - -
 stackoverflow.com/questions/8956331/how-to-get-results-from-the-wikipedia-api-with-phptext/..Testing for url -
wordpress
 goldendragonfamily.wordpress.comtext/..WordPress/4.0-alpha; url - -
 hoangkimlong.wordpress.comtext/..WordPress/4.0-alpha; url - -
 sdyankeereport.wordpress.comtext/..WordPress/4.0-alpha; url - -
 whatownsme.wordpress.comtext/..WordPress/4.0-alpha; url - -
 kingcrimsonprog.wordpress.comimage/..WordPress/4.0-alpha; url - -
 halokue.wordpress.comtext/..WordPress/4.0-alpha; url - -
github
 github.com/sparklemotion/mechanize/text/..Mechanize/2.7.2 Ruby/2.1.2p95 (url) en-us,en;q=0.5 -
 github.com/sparklemotion/mechanize/application/jsonMechanize/2.7.3 Ruby/2.1.0p0 (url) en-us,en;q=0.5 -
 github.com/pauldix/feedzirra/tree/masterapplication/xmlfeedzirra url - -
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url - -
 github.com/pauldix/feedzirra/tree/mastertext/..feedzirra url - -
flipboard
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; url) - -
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
 flipboard.com/browserproxytext/..null (FlipboardProxy/1.1; url) - -
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; url) en-us,en;q=0.5 -
example
 example.orgapplication/jsonArticle-Level Metrics 3.3-beta - url - -
 example.com/MyCoolToolPage/application/jsonMyCoolTool (url) - -
 example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - -
 example.orgapplication/jsonArticle-Level Metrics 3.2-beta - url - -
 alm.example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - -
wikilovesearth
 wikilovesearth.at/tools/text/..wle_at/1.0 (url; mail address ) - -
monitis
 www.monitis.comtext/..Mozilla/5.0 (compatible; monitis - premium monitoring service; url) - -
 monitis.comtext/..Mozilla/5.0 (compatible; monitis.com - free monitoring service; url) - -
xovibot
 www.xovibot.net/text/..Mozilla/5.0 (compatible; XoviBot/2.0; url) - -
 www.xovibot.net/image/..Mozilla/5.0 (compatible; XoviBot/2.0; url) - -
dynotes
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD (WP 7)/1.1.2 (url); mail address - -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address en-gb -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/4.9.2 (url; mail address en-us -
 www.dynotes.com/multi-lang-dictionary/application/jsonMLD/5.0.2 (url; mail address ru -
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url) - -
 help.zum.com/inquirytext/..Mozilla/5.0 (compatible; ZumBot/1.0; url) - -
diffbot
 www.diffbot.comimage/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) - -
 www.diffbot.comtext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (Diffbot/0.1; url) en-us,en;q=0.5 -
 www.diffbot.comimage/..Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; url) Chrome/24.0.1295.0 Safari/537.15 en-US,en;q=0.8 -
opensiteexplorer
 www.opensiteexplorer.org/dotbottext/..Mozilla/5.0 (compatible; DotBot/1.1; url, mail address ) - -
toolserver
 toolserver.org/~dispenser/image/..CacheThumbs/1.4 (url) - -
 toolserver.org/~dispenser/text/..DispensersTools (url) - -
 toolserver.org/~dispenser/text/..CacheThumbs/1.4 (url) - -
 toolserver.org/~erwin85/xwiki.phptext/..Wikipedia Bot - xwiki - url - -
crossref
 alm.labs.crossref.orgapplication/jsonArticle Level Metrics 3.1.1 - url -
aaronsw
 www.aaronsw.com/2002/feedfinder/text/..feedfinder/1.371 Python-urllib/1.16 url - -
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) en -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) - -
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.5; url) -
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0 ko-kr,ko;q=0.8,en-us;q=0.5,en;q=0.3 -
feedly
 www.feedly.com/fetcher.htmlapplication/xmlFeedly/1.0 (url; like FeedFetcher-Google) - -
 www.feedly.com/fetcher.htmltext/..Feedly/1.0 (url; like FeedFetcher-Google) - -
1eko
 shop1.maul.1eko.comtext/..WordPress/3.9.1; url - -
 shop1.tact.1eko.comtext/..WordPress/3.9.1; url - -
 shop1.match.1eko.comtext/..WordPress/3.9.1; url - -
 shop1.pard.1eko.comtext/..WordPress/3.9.1; url - -
 shop1.esaera.1eko.comtext/..WordPress/3.9.1; url - -
 shop1.magma.1eko.comtext/..WordPress/3.9.1; url - -
 online2.shopont.1eko.comtext/..WordPress/3.9.1; url - -
 shop1.oak.1eko.comtext/..WordPress/3.9.1; url - -
 online3.nioshop.1eko.comtext/..WordPress/3.9.1; url - -
 shop1.nab.1eko.comtext/..WordPress/3.9.1; url - -
 online3.oshop.1eko.comtext/..WordPress/3.9.1; url - -
 shop1.pal.1eko.comtext/..WordPress/3.9.1; url - -
 online2.evenshop.1eko.comtext/..WordPress/3.9.1; url - -
http:
 http://alm.example.orgapplication/jsonArticle-Level Metrics 3.1.1 - url - -
sanskritdictionary
 www.sanskritdictionary.com/application/vnd.php.serializedUser-Agent: SanskritDictionary/0.1 (url) - -
xbmc
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1;WOW64;Win64;x64; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.1; url) - -
 www.xbmc.orgimage/..XBMC/12.2 Git:20131005-4dd128f (Linux; Android; 3.0.50 armv7l; url) - -
 xbmc.orgimage/..XBMC/13.1 Git:20140604-84725b0 (Windows NT 6.3;WOW64;Win64;x64; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130430-d9119f2 (Linux; Android; 3.0.50 armv7l; url) - -
 www.xbmc.orgimage/..XBMC/12.0 Git:20130527-a36a92b (Linux; Android; 3.0.50 armv7l; url) - -
riddler
 riddler.io/about.htmltext/..Riddler (url) - -
007ac9
 crawler.007ac9.net/text/..Mozilla/5.0 (compatible; 007ac9 Crawler; url) - -
superfeedr
 superfeedr.comapplication/xmlSuperfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
 superfeedr.comtext/..Superfeedr bot/2.0 url - Make your feeds realtime: get in touch! - -
zeebox
 www.zeebox.comtext/..Zeebox (url) en-us,en;q=0.5 -
 www.zeebox.comapplication/jsonZeebox (url) en-us,en;q=0.5 -
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url) - -
wesee
 www.wesee.com/bot/image/..WeSEE:Ads/PictureBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PageBot (url) - -
 www.wesee.com/bot/text/..WeSEE:Ads/PictureBot (url) - -
phonifier
 www.phonifier.comtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
oncrawl
 www.oncrawl.com/bot.htmltext/..CdiscountBot-crawler-oncrawl/Nutch-1.7 (Mozilla/5.0 (compatible; CdiscountBot-crawler-oncrawl/0.1; url); http://www.oncrawl.com/bot.html; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
OpenLinkProfiler
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) en-us,en;q=0.5 -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) - -
 OpenLinkProfiler.org/bottext/..Mozilla/5.0 (compatible; spbot/4.1.0; url ) en-us,en;q=0.5
drupal
 drupal.org/text/..Drupal (url) - -
 drupal.org/application/jsonDrupal (url) - -
 drupal.org/image/..Drupal (url) - -
 drupal.org/text/..Drupal (url) -
 drupal.org/text/..User-Agent: Drupal (url) - -
muso
 www.muso.comtext/..Mozilla/5.0 (compatible; musobot/1.0; mail address ; url) - -
draigweb
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url - -
 www.draigweb.co.uk/wikitext/..WordPress/3.9; url -
archive-it
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url) * -
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot; url) - -
leiki
 www.leiki.comtext/..Leikibot/1.0 (url) - -
saciol
 www.saciol.com/text/..User-Agent: Saciolbot/1.2 (url; mail address - -
 www.saciol.com/application/jsonUser-Agent: Saciolbot/1.2 (url; mail address - -
bl
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmlimage/..bl.uk_lddc_bot/3.1.1 (url) - -
 www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.htmltext/..bl.uk_lddc_bot/3.1.1 (url) - -
kinshoo
 www.kinshoo.com/bot.htmltext/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 -
 kinshoo.net/bot.htmltext/..kinshoobot (/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.8.0_05; Europe/fr) url en-us,en;q=0.5 -
 kinshoo.net/bot.htmltext/..kinshoobot (/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_60; Europe/fr) url en-us,en;q=0.5 -
 kinshoo.net/bot.htmlimage/..kinshoobot (/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.7.0_60; Europe/fr) url en-us,en;q=0.5 -
 www.kinshoo.com/bot.htmlimage/..KiNShooboT (compatible; KiNShooboT/1.0.C; url) en-us,en;q=0.5 -
copernicus
 metricus.copernicus.orgapplication/jsonArticle Level Metrics - url - -
webmeup-crawler
 webmeup-crawler.com/text/..Mozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/application/jsonMozilla/5.0 (compatible; BLEXBot/1.0; url) - -
 webmeup-crawler.com/text/..(compatible; BLEXBot/1.0; url) - -
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
creativecommons
 wiki.creativecommons.org/Metadata_Scrapertext/..CC Metadata Scaper url - -
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url) -
 www.moviecus.com/botcontactinfo.phpapplication/jsonmoviecus bot (url) -
wikidict
 www.wikidict.detext/..url - -
okian
 www.okian.ro/text/..MyBot/1.0 (url) - -
 www.okian.ro/application/jsonMyBot/1.0 (url) - -
weblio
 www.weblio.jp/info/crawler.jspimage/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url) - -
 www.weblio.jp/info/crawler.jsptext/..Mozilla/5.0 (compatible; Webliobot/0.1; url) - -
embed
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url) - -
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url) - -
tineye
 www.tineye.com/crawler.htmltext/..TinEye-bot/0.02 (see url) - -
 www.tineye.com/crawler.htmlimage/..TinEye-bot/0.02 (see url) - -
pchome
 www.pchome.com.tw/pchomebot.htmtext/..Mozilla/5.0 (compatible; PChomebot/1.0; url) zh-tw,zh-cn,zh,en-us,en;q=0.7,*;q=0.3 -
grouphigh
 www.grouphigh.com/text/..Mozilla/5.0 (compatible; GroupHigh/1.0; url) - -
 www.grouphigh.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GroupHigh/1.0; url) - -
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
 www.trendiction.de/botimage/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 en-gb,en;q=0.5 -
zaigar
 lv.zaigar.infotext/..WordPress/3.2.1; url - -
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address ) - -
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url) - -
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url) - -
bit
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) de
 bit.ly/1dJuuzstext/..Seobility (SEO-Check; url) en
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url) en;q=0.8, * -
SearchNearMe
 SearchNearMe.com/contact.phptext/..SearchNearMe (url) - -
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url) - -
apache
 lucene.apache.org/nutch/bot.htmltext/..NutchCVS/0.7.2 (Nutch; url; mail address ) - -
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url) - -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) ja, en -
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;url) - -
mytest
 www.mytest.comtext/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
 www.mytest.comimage/..Mozilla/5.0 (compatible; heritrix/3.2.0 url) - -
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url) - -
netvibes
 www.netvibes.comtext/..Netvibes (url) - -
 www.netvibes.comapplication/jsonNetvibes (url) en-us -
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url) - -
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/3.0; url) en-gb,en;q=0.5 -
yoursite
 yoursite.com/botinfotext/..Mozilla/5.0 (compatible; YourCoolBot/1.0; url) - -
 text/..GoogleBot/1.0 (mail address color=red>GoogleBot.com urlGoogleBot.com/) de -
 image/..Mozilla/5.0 (compatible; GoogleBot/2.1; url www.google.com/ bot.html) en-US,en;q=0.5 -
 text/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) urlGoogleBot.com Gecko/20100101 Firefox/23.0 en-us,en;q=0.5 -
perfectmarket
 www.perfectmarket.comtext/..Mozilla/5.0 (compatible; PerfectMarketKWTbot/1.0 url) - -
test
 test.co.kr/text/..xxxxxxxx/Nutch-2.2.1 (url) - -
wymanenergy
 wymanenergy.com/ustext/..WordPress/3.9.1; url - -
 wymanenergy.com/uktext/..WordPress/3.9.1; url - -
 wymanenergy.com/estext/..WordPress/3.9.1; url - -
 wymanenergy.com/frtext/..WordPress/3.9.1; url - -
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
 spinn3r.com/robotapplication/xmlMozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19 - -
proximic
 www.proximic.com/info/spider.phptext/..Mozilla/5.0 (compatible; proximic; url) - -
simplepie
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103 - -
 simplepie.orgapplication/xmlSimplePie/1.3.1 (Feed Parser; url; Allow like Gecko) Build/20121030175403 - -
 simplepie.orgtext/..SimplePie/1.3.1 (Feed Parser; url; Allow like Gecko) Build/20121030175403 - -
blogspot
 beh1001.blogspot.com/p/the-bot.htmltext/..Mozilla/5.0 (compatible; Beh1001/beta; url) - -
 alberapps.blogspot.comtext/..TiempoBus/2.8 (url; mail address ) - -
gulliway
 gulliway.orgtext/..Mozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
 gulliway.orgapplication/xmlMozzila/5.0 (Windows NT 5.1; GulliwayBot/01 url) - -
Anonymouse
 Anonymouse.org/image/..url (Unix) - -
 Anonymouse.org/text/..url (Unix) - -
ac
 www.tkl.iis.u-tokyo.ac.jp/~crawler/text/..Mozilla/5.0 (compatible; Steeler/3.5; url) ja,en -
 www.clips.ua.ac.be/patternapplication/jsonPattern/2.6 url - -
 webarchiv.onb.ac.atimage/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
 webarchiv.onb.ac.attext/..Mozilla/5.0 (compatible; htrix/1.14.4 url) - -
2wap
 www.2wap.orgtext/..Mozilla/5.0 (compatible; Phonifier; url) - -
metamagazine
 metamagazine.comtext/..WordPress/3.9.1; url -
watchmouse
 watchmouse.com/image/..WatchMouse (url ; HQ) en-us,en;q=0.5 -
waw
 www.ipipan.waw.pl/nekst/nekstbot/text/..Mozilla/5.0 (Nekstbot; url) - -
vk
 vk.com/dev/Sharetext/..Mozilla/5.0 (compatible; vkShare; url) -
 vk.com/dev/Shareimage/..Mozilla/5.0 (compatible; vkShare; url) -
sentymetr
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url) - -
xxx
 www.xxx.comtext/..spider/Nutch-1.5.1 (spider; url) en-us,en-gb,en;q=0.7,*;q=0.3 -
crazywebcrawler
 www.crazywebcrawler.comtext/..CRAZYWEBCRAWLER 0.9.0, url - -
tt-rss
 tt-rss.org/application/xmlTiny Tiny RSS/1.12 (url) - -
MusicInTheRaw
 MusicInTheRaw.comtext/..MusicInTheRaw/0.1b (url; mail address ) using RestSharp/2.6' - -
linkdex
 www.linkdex.com/bots/text/..Mozilla/5.0 (compatible; linkdexbot/2.0; url) - -
twitter
 twitter.com/wiederfreitext/..wiederfreibot/1.0 (url) - -
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address ) - -
searchmetrics
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) - -
 www.searchmetrics.com/en/searchmetrics-bot/text/..Mozilla/5.0 (compatible; SearchmetricsBot; url) -
livedoor
 reader.livedoor.com/application/xmllivedoor FeedFetcher/0.01 (url; 1 subscriber) - -
stolencamerafinder
 www.stolencamerafinder.com/text/..SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
deepcrawl
 deepcrawl.co.uk/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url) - -
pingdom
 www.pingdom.com/text/..Pingdom.com_bot_version_1.4_(url) - -
 www.pingdom.comtext/..Pingdom.com_bot_version_1.4_(url) - -
inagist
 inagist.comtext/..InAGist URL Resolver (url) en-us,en;q=0.5 -
ireplay
 ireplay.fr/text/..iReplay/1.0 (url) -
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) en -
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url) tr -
mysite
 www.mysite.com/application/jsonMyBot/1.0 (url) - -
 www.mysite.com/botinfoapplication/vnd.php.serializedMozilla/5.0 (compatible; MyCoolBot/1.0; url) - -
comodorobot
 www.comodorobot.comtext/..Comodo-Webinspector-Crawler 2.2.1, url - -
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url) - -
 crawler.sistrix.net/text/..(compatible; SISTRIX Crawler; url)
cosmix
 www.cosmix.fr/application/jsonCosmix (url) - -
temnos
 temnos.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1 en-us,en;q=0.5 -
daringtodo
 www.daringtodo.comtext/..WordPress/3.8.3; url - -
edu
 www.aau.edu.ettext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url for Ethiopian Web archive project) - -
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url) - -
jetsli
 jetsli.de/crawlertext/..Mozilla/5.0 (compatible; Jetslide; url) en-us -
wikiapiary
 wikiapiary.com/wiki/User:Bumble_Beeapplication/jsonBumble Bee/1.0 (WikiApiary; url) -
parsijoo
 www.parsijoo.ir/text/..Mozilla/5.0 (compatible; parsijoo; url; mail address ) - -
creativepulses
 creativepulses.nltext/..CreativePulses Crawler (url) - -
fotopedia
 www.fotopedia.comapplication/jsonPicor (url) - -
mobileproxy
 mobileproxy.mobitext/..Mozilla/5.0 (compatible; MobileSurf; url) - -
tweetedtimes
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) - -
 tweetedtimes.comtext/..Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; url) en-US,en;q=0.8,fr;q=0.6,ru;q=0.4 -
gorgor
 gorgor.irtext/..hivaBot/hivaBot-1.0 (url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 gorgor.irtext/..hivaBot/hivaBot-1.0 (Iranian Search Engine bot; url; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
inoutscripts
 www.inoutscripts.com/bot.htmltext/..Mozilla/5.0 (compatible; Nesotebot/2.1; url) - -
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url) - -
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url) - -
softlist
 www.softlist.us/text/..Mozilla/5.0 (compatible; SoftListBot/2.2; url) en -
quaba
 quaba.detext/..quaba-spider (url) - -
aol-soft
 www.aol-soft.com/text/..Mozilla/5.0 (compatible; AOLbot/4.0; url) en -
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url) - -
 www.picsearch.com/bot.htmltext/..psbot-image (url) - -
189298.789999979total

Page requests for probable crawlers, recognized by keyword
Count
x 1000		Agent string
  Mime type (count ≥ 3)
pwb/rg3428 Pywikibot/2.0 - -
 application/json
Peachy MediaWiki Bot API Version 2.0 (alpha 8) - -
 application/vnd.php.serialized
 text/..
 application/json
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/pdf
 application/ogg
MediaWikiCrawler-Google/2.0 ( mail address ) - -
 text/..
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider zh-CN -
 text/..
 image/..
 application/ogg
 video/webm
 application/xml
 audio/webm
 audio/midi
php wikibot classes - -
 application/vnd.php.serialized
 text/..
py-radehh.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
 application/pdf
wikiwix-bot-3.0 - -
 text/..
LinkParser/2.0 - -
 text/..
ClueBot/2.0 - -
 application/vnd.php.serialized
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Facebot/1.0 -
 text/..
 image/..
 application/ogg
 application/xml
 application/pdf
 video/webm
 audio/midi
tigerbot - -
 application/json
 text/..
 image/..
 audio/x-flac
pywikipedia-newar4ur.py/r10843 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-rad1.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzfatranslator.py/r10843 Pywikipediabot/1.0 - -
 application/json
pwb/rg3113 Pywikipediabot/2.0 - -
 application/json
GoogleBot-Image/1.0 - -
 image/..
 text/..
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en,* -
 image/..
 text/..
 application/json
Twitterbot/1.0 - -
 text/..
 image/..
 application/ogg
 application/pdf
 application/json
maj_articles_recents/rg3489 Pywikibot/2.0 - -
 application/json
ProoXiBot fr -
 text/..
 application/ogg
 application/xml
Hoo Bot (https://meta.wikimedia.org/wiki/User:Hoo_man) - -
 application/json
 text/..
pywikibot-1newar25.py/rg11016 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-1newar25.py/rg11023 Pywikipediabot/1.0 - -
 application/json
 application/xml
PythonWikipediaBot/1.0 - -
 application/json
 text/..
BeneBot*/1.0 WikibasePhpLib/0.1 - -
 application/json
pywikibot-compat-redirect.py/rg10993 Pywikipediabot/1.0 - -
 application/json
MediaWiki::Bot/5.005006 - -
 application/json
Mozilla/5.0 (Windows NT 5.1; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address ) fr; q=1.0, en; q=0.5, *; q=0.1 -
 text/..
DigitalsmithsBot - -
 text/..
maj_articles_recents/rg3465 Pywikibot/2.0 - -
 application/json
Owlin Bot v. 3.0 - -
 text/..
MediaWiki::Bot/3.2.6 - -
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 KHTML Version/6.0 Mobile/10A5376e Safari/8536.25(compatible;SogouMobileSpider/2.0) zh-cn -
 text/..
 -
pywikipedia-radeh.py/rg10961 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
pywikibot-1newar25.py/rg11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
milog_bot/1.0 ( mail address ) - -
 text/..
AnomieBOT/1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
AniBot/0.9 php/curl - -
 application/vnd.php.serialized
YisouSpider zh-cn,en-us,zh-tw,en-gb,en; -
 text/..
 application/ogg
UAA-UAA.py/r10417 Pywikipediabot/1.0 - -
 application/json
mail address mail address – MediaWiki Tcl Bot Framework 1.0 - -
 application/json
pywikibot-1newar25.py/rg11025 Pywikipediabot/1.0 - -
 application/json
 application/xml
dikantenyvaovao/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Webbot - -
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
MediaWiki::Bot/3.005002 - -
 application/json
pywikipedia-redirect.py/r11781 Pywikipediabot/1.0 - -
 application/json
 text/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]]) - -
 application/json
spider - -
 text/..
 application/json
 application/vnd.php.serialized
 image/..
pywikipedia-zzfatranslator.py/rg10961 Pywikipediabot/1.0 - -
 application/json
pywikipedia-imageuncat.py/rg10972 Pywikipediabot/1.0 - -
 application/json
 application/xml
WikidataBot framework - -
 application/json
ContextAd Bot 1.0 - -
 text/..
 image/..
 application/pdf
 application/json
 application/ogg
Screaming Frog SEO Spider/2.50 - -
 text/..
 image/..
360spider-image - -
 image/..
 text/..
DotNetWikiBot/3.0 (Microsoft Windows NT 5.1.2600 Service Pack 3; ) - -
 text/..
pywikibot-compat-abi-17-progreso1911.py/rg10993 Pywikipediabot/1.0 - -
 application/json
 application/xml
GoogleBot-Image/1.0 - https=1
 text/..
Facebot/1.0 - -
 text/..
 application/json
 application/xml
 image/..
mgverbscript/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-1newar25.py/rg11018 Pywikipediabot/1.0 - -
 application/json
gsa-crawler (Enterprise; T2-KPTYDNBXJWWKK; mail address ) - -
 text/..
DotNetWikiBot/2.100 (Unix 3.2.0.59; ) - -
 text/..
 application/xml
pywikipedia-zz_make_cat.py/rg10961 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.99 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
Privacy Crawler (Delft University of Technology) - -
 text/..
www.integromedb.org/Crawler - -
 text/..
 application/xml
 application/json
 image/..
pywikipedia3-MakeScoreTable_test3.py/r11448 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/2.102 (Microsoft Windows NT 6.2.9200.0; ) - -
 text/..
 application/xml
GermCrawler - -
 application/json
 text/..
 application/xml
compat-featured.py/r10441 Pywikipediabot/1.0 - -
 application/json
 application/xml
python-wikitools/1.2 (User:DickensBot) -
 application/json
MediaWiki::Bot/5.005007 - -
 application/json
pywikibot-1newar25.py/rg11026 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-git-wdph111.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-welcome.py/r11252 Pywikipediabot/1.0 - -
 application/json
cwikt-interwiki.py/rg10992 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
HTMLParser/2.0 - -
 text/..
mijnbots-RedirDiakriet.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
Yeobot/1.1 Contact: mail address ) - -
 text/..
pwb/rg2925 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-aaa_elenco.py/rg10952 Pywikipediabot/1.0 - -
 application/json
pywikibot-1newar25.py/rg11028 Pywikipediabot/1.0 - -
 application/json
 application/xml
SecretaryBot (https://github/vlsergey/Secretary; mail address ) BasedOnJWBF - -
 text/..
 application/json
Crawler: https://github.com/wdenton/wikipedia.edits ( mail address ) - -
 application/json
Bot - -
 text/..
getpron/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
compat-featured.py/rg11005 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-zzinter.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-welcome-custom.py/r10249 Pywikipediabot/1.0 - -
 application/json
run_replace.7r/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikipedia-welcome.py/r11597 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 KHTML Chrome/21.0.1180.89 Safari/537.1; 360Spider - -
 text/..
 application/json
 application/xml
WikiTrans.net Bot (User:WikiTransBot; Contact: mail address ) - -
 text/..
 application/json
compat-checkimages.py/rs11005 Pywikipediabot/1.0 - -
 application/json
pwb/r2954 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-featured.py/r90 Pywikipediabot/1.0 - -
 application/json
MyCuteBot/0.1 - -
 text/..
 application/json
pywikipedia-lit.py/rg11023 Pywikipediabot/1.0 - -
 application/json
pwb/rs4505 Pywikibot/2.0 - -
 application/json
pywikibot-1newar25.py/rg11032 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-del_new.py/r90 Pywikipediabot/1.0 - -
 application/json
SineBot/1.5.19(User:SineBot) - -
 application/vnd.php.serialized
 text/..
rijksmonumenten_import/rg3397 Pywikibot/2.0 - -
 application/json
compat-redirect.py/rs10992 Pywikipediabot/1.0 - -
 application/json
 application/xml
getpron/r-1 (unknown) Pywikibot/2.0 - -
 application/json
Webwiki Search Engine Bot - www.webwiki.de - -
 text/..
ko_missing2/r-1 (unknown) Pywikibot/2.0 - -
 application/json
bot-05-links-to-labels/rg3433 Pywikibot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-basictemplatedisamblist.py/rg11016 Pywikipediabot/1.0 - -
 application/json
 text/..
 application/xml
CorenSearchBot/1.7 en libwww-perl/6.03 - -
 text/..
ITeco bot, v0.0, mail address - -
 text/..
KatBot bot.nytlabs.com> - -
 text/..
BOT/0.1 (BOT for JCE) - -
 text/..
ko_newpage/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-aaa_categorieAutori.py/rg10952 Pywikipediabot/1.0 - -
 application/json
 application/xml
codes_insee/r2554 Pywikipediabot/2.0 - -
 application/json
Python27-pythonw.exe/r(unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
pyiw-redirectns0.py/r11641 Pywikipediabot/1.0 - -
 application/json
WorldOfMusicBetaBot/1.0 - -
 text/..
python-wikitools/1.2 (User:Irclogbot) - -
 application/json
wikiparser/1 CFNetwork/673.4 Darwin/13.2.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
DotNetWikiBot/2.103 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
compat-basictemplatedisamblist.py/rg11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-welcome.py/r11622 Pywikipediabot/1.0 - -
 application/json
AnomieBOT/1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
pywikibot-compat-commonscat.py/rg11026 Pywikipediabot/1.0 - -
 application/json
 application/xml
bot-head.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
anagrama/r-1 (unknown) Pywikibot/2.0 - -
 application/json
pwb/rs4554 Pywikibot/2.0 - -
 application/json
pwb/rg3536 Pywikibot/2.0 - -
 application/json
pywikibot-compat-commonscat.py/rg11028 Pywikipediabot/1.0 - -
 application/json
My Nutch Spider/Nutch-1.7 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
BotReversor-BotReversor.py/rg11003 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-coords.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-featured.py/r(unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
wikiparser/1 CFNetwork/673.4 Darwin/13.3.0 (x86_64) (MacPro5,1) en-us -
 image/..
 text/..
py-navbox.py/rg10975 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-checkimages.py/rs11002 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (compatible; SemanticBot/1.1) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikipedia3-MakeScoreTable_10000.py/r11448 Pywikipediabot/1.0 - -
 application/json
AnomieBOT/1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
pywikibot-1newar25.py/rg11017 Pywikipediabot/1.0 - -
 application/json
SentiBot www.sentibot.eu (compatible with GoogleBot) - -
 text/..
svnpywikipedia-interwiki.py/r11270 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/rg11025 Pywikipediabot/1.0 - -
 application/json
Goalkeeperbot(User:Beetstra)/1.0 - -
 text/..
pwb/rs4600 Pywikibot/2.0 - -
 application/json
logpatrol/rg3280 Pywikibot/2.0 - -
 application/json
dikantenyvaovao/r-1 (unknown) Pywikibot/2.0 - -
 application/json
bot-05-links-to-labels/rg3467 Pywikibot/2.0 - -
 application/json
pywikipedia-git-wdlabel.py/r90 Pywikipediabot/1.0 - -
 application/json
PythonWikipediaBot/1.0 -
 application/json
 text/..
pywikipedia-git-maghalesaz2.py/r90 Pywikipediabot/1.0 - -
 application/json
pywikipedia-git-maghalesaz.py/r90 Pywikipediabot/1.0 - -
 application/json
XLinkBot/1.00 - -
 text/..
UKBot [[:no:Bruker:UKBot]] MwClient/0.6.6 (https://github.com/mwclient/mwclient) - -
 application/json
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]]) - -
 application/json
 text/..
iw2data/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.1.15 Darwin/14.0.0 en-us -
 image/..
 text/..
YisouSpider - -
 text/..
 application/json
maj_articles_recents/rg3608 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
pywikibot-compat-reportUsers.py/r10494 Pywikipediabot/1.0 - -
 application/json
pywikipedia-radeh2.py/rg10961 Pywikipediabot/1.0 - -
 application/json
SurakWare MediaWiki Bot/1.0 - -
 text/..
wAPI/1.1.1 (Bot: Cyberbot I Operator: Cyberpower678) - -
 application/vnd.php.serialized
bot-05-links-to-labels/rg3477 Pywikibot/2.0 - -
 application/json
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]]) - -
 application/json
cwikt-interwiki.py/rg11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pyiw-redirectns1.py/r11641 Pywikipediabot/1.0 - -
 application/json
User:WP 1.0 bot (operated by User:Theopolisme on enwiki) - -
 text/..
pwb/rs4615 Pywikibot/2.0 - -
 application/json
German Wikipedia Broken Weblinks Bot; contact: mail address - -
 text/..
 application/ogg
rijksmonumenten_import/rg3590 Pywikibot/2.0 - -
 application/json
DotNetWikiBot/2.100 (Unix 3.2.0.60; ) - -
 text/..
 application/xml
TwynCatBot/0.2 (Contact: www.twyn.com) - -
 application/json
botjagwar-getpron.py/r11215 Pywikipediabot/1.0 - -
 application/json
compat-featured.py/rg11027 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/r-1 (unknown) Pywikibot/2.0 - -
 application/json
ko_missing/r-1 (unknown) Pywikibot/2.0 - -
 application/json
MakeScoreTable/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DotNetWikiBot/2.101 (Unix 3.1.9.0; ) - -
 text/..
 application/xml
pywikibot-1newar25.py/rg11030 Pywikipediabot/1.0 - -
 application/json
unknown Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails) en;q=0.9,*;q=0.8 -
 text/..
 image/..
 application/ogg
AnomieBOT/1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
compat-featured.py/rg11025 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rs4538 Pywikibot/2.0 - -
 application/json
pwb/rs4535 Pywikibot/2.0 - -
 application/json
Webwiki Search Engine Bot - www.webwiki.de -
 text/..
pywikibot-1newar25.py/rg11029 Pywikipediabot/1.0 - -
 application/json
pywikipedia-wikidata_coordinaten_dms2.py/r11200 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywiki-zzCityLocation.py/rg11025 Pywikipediabot/1.0 - -
 application/json
 application/xml
User-Agent Baiduspider - -
 text/..
 image/..
bot-05-links-to-labels/rg3473 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1 - -
 image/..
 text/..
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
Mozilla/5.0 (compatible;acapbot/0.1.;treat like GoogleBot) en-us,en-gb;q=0.7,en;q=0.3 -
 text/..
 -
g13bot_tools-g13_nudge_bot.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-reportWBS.py/r10494 Pywikipediabot/1.0 - -
 application/json
Peachy MediaWiki Bot API Version 2.0 (alpha 7) - -
 application/vnd.php.serialized
pywikipedia-interwiki.py/rg8 Pywikipediabot/1.0 - -
 application/json
 application/xml
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]]) - -
 application/json
AnomieBOT/1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pwb/rs4595 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (compatible; MyBot/1.0;) -
 application/json
bot-nullref.py/r11330 (wikipedia.py) Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
Livelapbot/0.1 en-US,en -
 text/..
 image/..
 application/pdf
 application/ogg
Google Bot - -
 application/json
 text/..
compat-basicarticledisamblist.py/rg11027 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-aaa_categorieTesti.py/rg10952 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-interwiki.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
t9/rg2816 Pywikipediabot/2.0 - -
 application/json
pywikibot-compat-checkimages.py/rg10976 Pywikipediabot/1.0 - -
 application/json
pywikibot-compat-commonscat.py/rg11016 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; de-de; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; es-es; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
 application/json
porzucone/rg3489 Pywikibot/2.0 - -
 application/json
compat-basicarticledisamblist.py/rg11026 Pywikipediabot/1.0 - -
 application/json
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]]) - -
 application/json
pywikipedia-zzaddnavebox.py/rg10961 Pywikipediabot/1.0 - -
 application/json
pwb/rs4479 Pywikibot/2.0 - -
 application/json
MediaWiki::Bot/3.1.6 (User:SporkBot) - -
 application/json
2552753/rg3563 Pywikibot/2.0 - -
 application/json
run_pagefromfile.2/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
porzucone/rg3609 Pywikibot/2.0 - -
 application/json
compat-basictemplatedisamblist.py/rg11026 Pywikipediabot/1.0 - -
 application/json
pywiki-zzCityLocation.py/rg11026 Pywikipediabot/1.0 - -
 application/json
 application/xml
pwb/rg3501 Pywikibot/2.0 - -
 application/json
conta/r-1 (unknown) Pywikibot/2.0 - -
 application/json
pywikipedia-rad1.py/rg10908 Pywikipediabot/1.0 - -
 application/json
MPDP-ALR-Search-Bot - -
 text/..
 image/..
porzucone/rg3487 Pywikibot/2.0 - -
 application/json
pwb/rg3461 Pywikibot/2.0 - -
 application/json
pywikibot-compat-pwb.py/rg10940 Pywikipediabot/1.0 - -
 application/json
 application/xml
 text/..
rijksmuseum_import /rg3397 Pywikibot/2.0 - -
 application/json
pro/rg3544 Pywikibot/2.0 - -
 application/json
refreshfeeds/rg3487 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
porzucone/rg3477 Pywikibot/2.0 - -
 application/json
pywikibot-compat-sichteroptin.py/r10883 Pywikipediabot/1.0 - -
 text/..
 application/json
pywikibot-compat-commonscat.py/rg11032 Pywikipediabot/1.0 - -
 application/json
pwb/rg3599 Pywikibot/2.0 - -
 application/json
pywiki-cfd.py/rg10926 Pywikipediabot/1.0 - -
 application/json
 application/xml
porzucone/rg3465 Pywikibot/2.0 - -
 application/json
Doc mail address mail address – MediaWiki Tcl Bot Framework 1.0 - -
 application/json
election/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (compatible; GoogleBot/2.1;) en-US,en;q=0.5 -
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; en-us; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
 -
 application/json
pywikibot-compat-commonscat.py/rg11013 Pywikipediabot/1.0 - -
 application/json
pyiw-category2.py/r11641 Pywikipediabot/1.0 - -
 application/json
Wikibot/2.1.1 CFNetwork/672.1.14 Darwin/14.0.0 en-us -
 image/..
 text/..
Peachy MediaWiki Bot API Version 1.0 - -
 application/vnd.php.serialized
compat-featured.py/rg11028 Pywikipediabot/1.0 - -
 application/json
 application/xml
GoogleBot-Image/1.0 bg -
 text/..
 image/..
pwb/rs4611 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (compatible; UnisterBot; mail address ) de-DE;q=0.9,de;q=0.8,en;q=0.7,* -
 text/..
GoogleBot-Video/1.0 - -
 text/..
 application/ogg
 audio/webm
ArchiveTeam ArchiveBot/20140715.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 KHTML Chrome/35.0.1916.153 Safari/537.36 - -
 image/..
 text/..
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
bot-05-links-to-labels/rg3381 Pywikibot/2.0 - -
 application/json
pywiki-newar25.py/rg11028 Pywikipediabot/1.0 - -
 application/json
pywikipedia-zzpurging.py/rg10961 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
 application/json
pwb/rs4543 Pywikibot/2.0 - -
 application/json
pwb/rg3397 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: mail address ) - -
 text/..
 image/..
 application/pdf
 application/ogg
 video/webm
Erel Bot - -
 text/..
JackBot-fr.wikt.RemiseEnForme.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
addDescOrt/r2916 Pywikipediabot/2.0 - -
 application/json
compat-featured.py/rg11032 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia-dobrodoslice.py/r11775 Pywikipediabot/1.0 -
 application/json
pywikipedia-bot_rss.py/rg14302 Pywikipediabot/1.0 - -
 application/json
RSS Bot/1.8 (Mac OS X Version 10.9.4 (Build 13E28)) en;q=1, ja;q=0.9, fr;q=0.8, de;q=0.7, es;q=0.6, it;q=0.5 -
 application/xml
 -
My Bot - -
 text/..
compat-featured.py/rg11026 Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikibot-compat-commonscat.py/rg11023 Pywikipediabot/1.0 - -
 application/json
pywiki-newar25.py/rg11026 Pywikipediabot/1.0 - -
 application/json
pywikilab-llista_mils.py/rg11003 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3477 Pywikibot/2.0 - -
 application/json
compat-checkimages.py/rs11009 Pywikipediabot/1.0 - -
 application/json
pywiki-lit.py/rg11026 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3487 Pywikibot/2.0 - -
 application/json
pwb/rs4537 Pywikibot/2.0 - -
 application/json
pwb/rs4598 Pywikibot/2.0 - -
 application/json
articles/r2554 Pywikipediabot/2.0 - -
 application/json
compat-basicarticledisamblist.py/rg11018 Pywikipediabot/1.0 - -
 application/json
DotNetWikiBot/3.10 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
compat-featured.py/rg11016 Pywikipediabot/1.0 - -
 application/json
 application/xml
compat-featured.py/rg11030 Pywikipediabot/1.0 - -
 application/json
pywiki-lit.py/rg11025 Pywikipediabot/1.0 - -
 application/json
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) - -
 text/..
 image/..
update-task-categories/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 text/..
 -
pwb/rs4586 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; it-it; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
 application/json
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
py-new_iw6.py/rs11002 Pywikipediabot/1.0 - -
 application/json
python-wikitools/1.2 (User:DickensBot) - -
 application/json
Mozilla/5.0 (compatible; SputnikImageBot/2.3) - -
 image/..
 text/..
refreshfeeds/rg3489 Pywikibot/2.0 - -
 application/json
pywikipedia-com_cat.py/r10809 Pywikipediabot/1.0 - -
 application/json
Crawler/1.2 - -
 text/..
Nimit is a web indexing system designed to focus crawl the web for educational content. the crawler is built on top of open-source nutch and is intended to to pocketize the web into domains eg content for engineering, medicine,law, humanities and business/Nutch-1.6 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
 image/..
 application/ogg
test/r-1 (unknown) Pywikibot/2.0 - -
 application/json
PureBasic WikiBotFramework - d:User:InkoBot operated by d:User:Inkowik - -
 text/..
DotNetWikiBot/2.102 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
crawler4j for XQuery -
 text/..
run/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
mail address - -
 application/json
 text/..
GoogleBot-Image/1.0 -
 text/..
 image/..
Screaming Frog SEO Spider/2.40 - -
 text/..
 application/json
 image/..
Wegtam Crawler/Nutch-1.9-SNAPSHOT en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT ONE Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 text/..
 -
 application/json
MoodleBot/1.0 - -
 application/xml
 text/..
 application/vnd.php.serialized
 image/..
 application/ogg
refreshfeeds/rg3477 Pywikibot/2.0 - -
 application/json
MFU Spider(167)/Nutch-2.2.1 - -
 text/..
RBot/0.3 ( mail address ) -
 application/json
 image/..
 text/..
erfgoedbot-categorize_images.py/rg10979 Pywikipediabot/1.0 - -
 application/json
 application/xml
Kolega2357-replace.py/r-1 (unknown) Pywikipediabot/1.0 - -
 application/json
 application/xml
pywikipedia2-welcome.py/r10550 Pywikipediabot/1.0 - -
 application/json
pwb/rs4639 Pywikibot/2.0 - -
 application/json
pywikipedia-newinterwiki-fa.py/rg11028 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3526 Pywikibot/2.0 - -
 application/json
porzucone/rg3569 Pywikibot/2.0 - -
 application/json
wikibot-new-fixing_redirects.py/rg10986 Pywikipediabot/1.0 - -
 application/json
 application/xml
MpDmPrA-Search-Bot - -
 text/..
RokaSpider/Nutch-1.8 (' mail address ') de-de,de,en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko, Robot by TuiNu - -
 text/..
pwb/rs4636 Pywikibot/2.0 - -
 application/json
iw_labelcheck/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
DeclensionParser/0.02 ( mail address ) - -
 text/..
bot-05-links-to-labels/rg3483 Pywikibot/2.0 - -
 application/json
web-crawler en-US,en;q=0.8 -
 text/..
porzucone/rg3599 Pywikibot/2.0 - -
 application/json
porzucone/rg3471 Pywikibot/2.0 - -
 application/json
read_ehak/rg3297 Pywikibot/2.0 - -
 application/json
compat-featured.py/rg11017 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3483 Pywikibot/2.0 - -
 application/json
clean-training-set/rg3583 Pywikibot/2.0 - -
 application/json
porzucone/rg3507 Pywikibot/2.0 - -
 application/json
cd-delinker.py/r11775 Pywikipediabot/1.0 - -
 application/json
archiv-erl-bot/rg2953 Pywikipediabot/2.0 - -
 application/json
compat-noreferences.py/rg11016 Pywikipediabot/1.0 - -
 application/json
 application/xml
Mozilla/5.0 (Linux; Android/4.2.2; de-de; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 de-DE, en-US -
 image/..
 text/..
 -
DotNetWikiBot/2.103 (Microsoft Windows NT 6.0.6002 Service Pack 2; ) - -
 text/..
 application/xml
porzucone/rg3510 Pywikibot/2.0 - -
 application/json
porzucone/rg3587 Pywikibot/2.0 - -
 application/json
porzucone/rg3484 Pywikibot/2.0 - -
 application/json
porzucone/rg3511 Pywikibot/2.0 - -
 application/json
gsa-crawler (Enterprise; T4-N4DP54QDKA66K; mail address ) - -
 text/..
refreshfeeds/rg3465 Pywikibot/2.0 - -
 application/json
GoogleBot - -
 text/..
Mozilla/5.0 (Linux; Android/4.2.1; it-it; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 it-IT, en-US -
 image/..
 -
 text/..
porzucone/rg3590 Pywikibot/2.0 - -
 application/json
porzucone/rg3604 Pywikibot/2.0 - -
 application/json
refreshfeeds/rg3486 Pywikibot/2.0 - -
 application/json
Facebot/1.0 - https=1
 text/..
Mozilla/5.0 (compatible; MerchantCentricBot/1.0) en-US,* -
 text/..
porzucone/rg3473 Pywikibot/2.0 - -
 application/json
porzucone/rg3549 Pywikibot/2.0 - -
 application/json
CheburashkaSearchBot - -
 text/..
 image/..
pywiki-lit.py/rg11028 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3573 Pywikibot/2.0 - -
 application/json
plantilla_BDT/r-1 (unknown) Pywikibot/2.0 - -
 application/json
porzucone/rg3467 Pywikibot/2.0 - -
 application/json
rahvaarv/rg3297 Pywikibot/2.0 - -
 application/json
replace_template/r-1 (unknown) Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; CUBOT GT99 Build/JOP40D) AppleWebKit/535.19 KHTML Chrome/18.0.1025.166 Mobile Safari/535.19 es-ES,es;q=0.8 -
 image/..
 -
 text/..
Inlibris.com XMLBot/1.0 - -
 text/..
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: mail address ) - -
 text/..
COIBot/2.0 - -
 text/..
compat-basicdisambrefslist.py/rg11016 Pywikipediabot/1.0 - -
 application/json
 application/xml
ClueBot/2.0 (ClueBot NG Report Interface) - -
 text/..
UniversalFeedParser/5.1.3 https://code.google.com/p/feedparser/ - -
 application/xml
 text/..
py-interwiki.py/rs11006 Pywikipediabot/1.0 - -
 application/json
 application/xml
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 KHTML MsnBot-Media /1.0b" en-US -
 image/..
 text/..
 application/json
Mozilla/5.0 (compatible; Mojeekbot/0.6; https://www.mojeek.com/bot.html) - -
 text/..
pywikipedia-zzcity_article.py/rg10961 Pywikipediabot/1.0 - -
 application/json
 application/xml
My Nutch Spider/Nutch-1.5.1 en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
Python27-featured.py/r(unknown) Pywikipediabot/1.0 - -
 application/json
compat-checkimages.py/rs10992 Pywikipediabot/1.0 - -
 application/json
porzucone/rg3546 Pywikibot/2.0 - -
 application/json
GoogleBot-Image/1.0 zh-CN,zh;q=0.8,en;q=0.6 -
 image/..
 text/..
pwb/r3152 Pywikipediabot/2.0 - -
 application/json
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 - -
 image/..
 text/..
porzucone/rg3515 Pywikibot/2.0 - -
 application/json
porzucone/rg3581 Pywikibot/2.0 - -
 application/json
porzucone/rg3509 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.1; en-us; CUBOT GT99 Build/JOP40D) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 en-US -
 image/..
 -
 text/..
 application/json
AnomieBOT/1.0 (BAGBot; see [[User:AnomieBOT]]) - -
 application/json
 text/..
refreshfeeds/rg3473 Pywikibot/2.0 - -
 application/json
NITRC Nutch Spider/Nutch-1.6 (Neuroimaging Clearinghouse; www.nitrc.org; mail address ) en-us,en-gb,en;q=0.7,*;q=0.3 -
 text/..
pywikibot-compat-replace.py/rg10993 Pywikipediabot/1.0 - -
 application/json
 application/xml
SiocWikiBot/0.1 - -
 text/..
porzucone/rg3562 Pywikibot/2.0 - -
 application/json
porzucone/rg3572 Pywikibot/2.0 - -
 application/json
wymowa/rg3573 Pywikibot/2.0 - -
 application/json
wymowa/rg3569 Pywikibot/2.0 - -
 application/json
compat-basicarticledisamblist.py/rg11017 Pywikipediabot/1.0 - -
 application/json
refreshfeeds/rg3467 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; fr-fr; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 fr-FR, en-US -
 image/..
 text/..
 -
 application/json
Bub's wikibot (Wikibot/2011111111; JWBF/1.2; Java/1.8) - -
 text/..
pywikipedia-zzfawikipatroler.py/rg10961 Pywikipediabot/1.0 - -
 application/json
COIBot/1.00 - -
 text/..
wymowa/rg3543 Pywikibot/2.0 - -
 application/json
bot-05-links-to-labels/rg3378 Pywikibot/2.0 - -
 application/json
pywiki-Modon1.py/rg11028 Pywikipediabot/1.0 - -
 application/json
 application/xml
HTMLParser/1.6 - -
 text/..
TFAdefenderbot/rg3077 Pywikipediabot/2.0 - -
 application/json
refreshfeeds/rg3483 Pywikibot/2.0 - -
 application/json
recent/r3183 Pywikibot/2.0 - -
 application/json
maj_articles_recents/rg3434 Pywikibot/2.0 - -
 application/json
Mozilla/5.0 (Linux; Android/4.2.2; es-es; CUBOT GT72E Build/JDQ39) AppleWebKit/534.30 KHTML Version/4.0 Mobile Safari/534.30 es-ES, en-US -
 image/..
 -
 text/..
 application/json
Screaming Frog SEO Spider/2,50 - -
 text/..
 image/..
wymowa/rg3590 Pywikibot/2.0 - -
 application/json
OpenSearchServer_Bot - -
 text/..
 image/..
MyFirstBot/0.01 - -
 application/vnd.php.serialized
wymowa/rg3515 Pywikibot/2.0 - -
 application/json
wymowa/rg3600 Pywikibot/2.0 - -
 application/json
wikibot-wpne-fixing_redirects.py/rg10986 Pywikipediabot/1.0 - -
 application/json
 application/xml
wymowa/rg3483 Pywikibot/2.0 - -
 application/json
wymowa/rg3589 Pywikibot/2.0 - -
 application/json
pwb/rg2800 Pywikipediabot/2.0 - -
 application/json
pywikipedia-git-new_iw14.py/r90 Pywikipediabot/1.0 - -
 application/json
AnomieBOT/1.0 (RandomPagePicker; see [[User:AnomieBOT]]) - -
 application/json
wymowa/rg3608 Pywikibot/2.0 - -
 application/json
IssueCrawler - -
 text/..
pywikipedia-zzredirectyeh.py/r10582 Pywikipediabot/1.0 - -
 application/json
 application/xml
2143150/rg3478 Pywikibot/2.0 - -
 application/json
template/r-1 (unknown) Pywikipediabot/2.0 - -
 application/json
wymowa/rg3617 Pywikibot/2.0 - -
 application/json
wikibot-wpne-welcome.py/rg10986 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3465 Pywikibot/2.0 - -
 application/json
DotNetWikiBot/3.0 (Microsoft Windows NT 6.1.7601 Service Pack 1; ) - -
 text/..
 application/xml
wymowa/rg3549 Pywikibot/2.0 - -
 application/json
pywikipedia-newar.py/rg11023 Pywikipediabot/1.0 - -
 application/json
wymowa/rg3473 Pywikibot/2.0 - -
 application/json
wymowa/rg3486 Pywikibot/2.0 - -
 application/json
37798.55total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Mon, Aug 4, 2014 10:39
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers