Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Jun 2011 - 30 Jun 2011

 This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
 See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

 
This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.

Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.


In total Google was somehow involved in 40.9% of daily external page* requests

Google referred to our sites, through its services including search, maps, and Google Earth, 139,448,000 page views per day, representing 35.5% of our external page requests.

Including all of its different search crawlers and services hosted on its servers, Google itself requested another 21,378,000 page pages per day, representing 5.4% of our external page requests.

* = mime type text/html only

In order of request volume

Requests originating from a Google ip address
ServiceTotalPagesImagesOther
GoogleBot17,48315,611928944
Wireless3,7133,645644
Other2,4161,0153981,003
Translate1,0061,00511
FeedFetcher3578928240
Web search15131 
Image search11  
Toolbar11  
Desktop1  1
Maps1 1 
Total24,99021,3781,4192,191
 
Requests originating from elsewhere
ServiceTotalPagesImagesOther
Web search136,104132,2512,5491,304
GoogleBot?7,5606,90038621
Desktop76721 746
Other478131101246
Maps162331281
Toolbar138511571
Image search5530251
Earth5222301
Translate15492
Mail7251
Wireless2111
FeedFetcher2211
Total145,342139,4482,9012,991
 
Top level domains
.com46,85345,2761,237340
.de8,6448,5239131
.uk8,0807,89615034
.jp7,3986,77543579
.fr5,7225,58411523
.it5,4165,28410923
.br5,1555,0449021
.ca4,1394,0487220
.es3,6943,5839120
.pl3,5173,4337212
.in3,2533,1028962
.au2,8782,8155310
.ru2,0571,987628
.mx1,9801,936359
.nl1,8841,830486
.ph1,4461,412269
.ar1,1681,136257
.be1,1271,097274
.tr1,1241,087298
.ch1,0301,001254
.th1,0169614412
.se945925172
.at915889224
.ua825809124
.cl774753174
.pt703685162
.pe684660168
.hk6726331624
.id666627354
.co640627112
.fi626613111
.ve619605104
.il611594125
.my603581184
.hu573550203
.ro563543173
.sg49648663
.vn4964632111
.nz49048191
.dk483473101
.gr450437112
.cz437419171
.ie43142542
.no423411111
.tw40238985
.bg37736792
.pk303287106
.za29228282
.hr27826982
.sk25224291
.eg21620295
.sa20119362
.ma17817143
.lt17717161
.org17641711
.kr165145119
.ec16215831
.ae15815531
.si15715251
.gt10610222
.ba1029741
.by1019911
.do1019911
.uy949221
.ee878521
.lv858231
.cr848221
.pr848221
.ng747112
.lk736841
.bo696621
.sv616011
.ke595811
.tn575521
.rs575511
.jo504821
.tt494811
.md474611
.py444311
.ge434021
.kz414111
.lu403911
.qa403911
.jm393811
.hn363511
.kw353411
.mt333311
.az302811
.lb292711
.is282811
.bh26251 
.gh262511
.np252311
.mu242411
.net243191
.ni232311
.am212011
.ps201911
.ws20120 
.iq191721
.pa191811
.om171711
.ug151411
.tz151511
.cm131211
.ci121211
.sn121111
.bn12111 
.mv111011
.mz111111
.et111111
.mn10911
.cat101011
.zw9911
.kh8711
.bs8811
.gp8711
.mg7611
.cu771 
.dm6611
.na661 
.kg6611
.uz6611
.rw5411
.dz5411
.bw5511
.gy5511
.cd4311
.bz441 
.ht44  
.zm44 1
.bd3311
.af3311
.ag331 
.ml3311
.vc3311
.biz2111
.bf22 1
.ao22 1
.ga2111
.gi22  
.la2211
.fj221 
.li221 
.vi22  
.mw22  
.eu2121
.mk221 
.ne11  
.ls111 
.tm111 
.ly111 
.ai111 
.me1111
.vg11  
.ir111 
.sl11  
.us1111
.bi11  
.as11  
.sc111 
.bj111 
.gm11  
.to1111
.cc111 
.tl111 
.fm111 
.ad111 
.tg11 1
.sm111 
.dj111 
.gg111 
.ki11  
.gl11  
.nu111 
.st111 
.hm11  
.ck11  
.ms11  
.pn11  
.sb1111
.gs1 1 
.cg11  
.tv1 1 
.tk111 
.su1 1 
.cn1111
.cx1 1 
.nr11  
.vu11  
.edu1 11
.ac11  
.je11  
.sh1 1 
.so11 1
.tj11  
.cf11  
.im111 
.mil11  
undefined32,68727,8211,0763,790
Total170,330160,8234,3205,179

In alphabetical order

Requests originating from a Google ip address
ServiceTotalPagesImagesOther
Wireless3,7133,645644
Web search15131 
Translate1,0061,00511
Toolbar11  
Other2,4161,0153981,003
Maps1 1 
Image search11  
GoogleBot17,48315,611928944
FeedFetcher3578928240
Desktop1  1
Total24,99021,3781,4192,191
 
Requests originating from elsewhere
ServiceTotalPagesImagesOther
Wireless2111
Web search136,104132,2512,5491,304
Translate15492
Toolbar138511571
Other478131101246
Maps162331281
Mail7251
Image search5530251
GoogleBot?7,5606,90038621
FeedFetcher2211
Earth5222301
Desktop76721 746
Total145,342139,4482,9012,991
 
Top level domains
.ac11  
.ad111 
.ae15815531
.af3311
.ag331 
.ai111 
.am212011
.ao22 1
.ar1,1681,136257
.as11  
.at915889224
.au2,8782,8155310
.az302811
.ba1029741
.bd3311
.be1,1271,097274
.bf22 1
.bg37736792
.bh26251 
.bi11  
.biz2111
.bj111 
.bn12111 
.bo696621
.br5,1555,0449021
.bs8811
.bw5511
.by1019911
.bz441 
.ca4,1394,0487220
.cat101011
.cc111 
.cd4311
.cf11  
.cg11  
.ch1,0301,001254
.ci121211
.ck11  
.cl774753174
.cm131211
.cn1111
.co640627112
.com46,85345,2761,237340
.cr848221
.cu771 
.cx1 1 
.cz437419171
.de8,6448,5239131
.dj111 
.dk483473101
.dm6611
.do1019911
.dz5411
.ec16215831
.edu1 11
.ee878521
.eg21620295
.es3,6943,5839120
.et111111
.eu2121
.fi626613111
.fj221 
.fm111 
.fr5,7225,58411523
.ga2111
.ge434021
.gg111 
.gh262511
.gi22  
.gl11  
.gm11  
.gp8711
.gr450437112
.gs1 1 
.gt10610222
.gy5511
.hk6726331624
.hm11  
.hn363511
.hr27826982
.ht44  
.hu573550203
.id666627354
.ie43142542
.il611594125
.im111 
.in3,2533,1028962
.iq191721
.ir111 
.is282811
.it5,4165,28410923
.je11  
.jm393811
.jo504821
.jp7,3986,77543579
.ke595811
.kg6611
.kh8711
.ki11  
.kr165145119
.kw353411
.kz414111
.la2211
.lb292711
.li221 
.lk736841
.ls111 
.lt17717161
.lu403911
.lv858231
.ly111 
.ma17817143
.md474611
.me1111
.mg7611
.mil11  
.mk221 
.ml3311
.mn10911
.ms11  
.mt333311
.mu242411
.mv111011
.mw22  
.mx1,9801,936359
.my603581184
.mz111111
.na661 
.ne11  
.net243191
.ng747112
.ni232311
.nl1,8841,830486
.no423411111
.np252311
.nr11  
.nu111 
.nz49048191
.om171711
.org17641711
.pa191811
.pe684660168
.ph1,4461,412269
.pk303287106
.pl3,5173,4337212
.pn11  
.pr848221
.ps201911
.pt703685162
.py444311
.qa403911
.ro563543173
.rs575511
.ru2,0571,987628
.rw5411
.sa20119362
.sb1111
.sc111 
.se945925172
.sg49648663
.sh1 1 
.si15715251
.sk25224291
.sl11  
.sm111 
.sn121111
.so11 1
.st111 
.su1 1 
.sv616011
.tg11 1
.th1,0169614412
.tj11  
.tk111 
.tl111 
.tm111 
.tn575521
.to1111
.tr1,1241,087298
.tt494811
.tv1 1 
.tw40238985
.tz151511
.ua825809124
.ug151411
.uk8,0807,89615034
.us1111
.uy949221
.uz6611
.vc3311
.ve619605104
.vg11  
.vi22  
.vn4964632111
.vu11  
.ws20120 
.za29228282
.zm44 1
.zw9911
undefined32,68727,8211,0763,790
Total170,330160,8234,3205,179
 

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted

Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.

Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)
 
ServiceTotalOriginating from
Google ip address
Referer mentions
Google url
Agent mentions
Google service
Desktop767--Y
Earth52--Y
FeedFetcher357Y-Y
GoogleBot17,481Y-Y
GoogleBot?7,490--Y
GoogleBot?69-YY
Image search55-Y-
Mail7-Y-
Maps162-Y-
Other354--Y
Other36-Y-
Other88-YY
Other234Y--
Other2,180Y-Y
Toolbar90--Y
Toolbar32-Y-
Toolbar16-YY
Translate13-Y-
Translate806Y-Y
Translate199YYY
Web search136,104-Y-
Web search15YY-
Wireless3,713Y-Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)

Note: averages below 1 are always rounded up to 1

Generated on Fri, Jul 15, 2011 22:17
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.