Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Nov 2010 - 30 Nov 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 44.5% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 157,527,000 page views per day, representing 41.5% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 11,702,000 page pages per day, representing 3.1% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 6,915 6,059 399 457

Wireless 3,918 3,858 57 4

Other 1,218 658 222 339

Translate 1,026 1,026 1 1

FeedFetcher 304 88 11 204

Web search 14 13 1 1

Toolbar 1 1

Desktop 1 1

Maps 1 1 1

Total 13,395 11,702 689 1,005

Requests originating from elsewhere

Service Total Pages Images Other

Web search 159,732 154,110 2,842 2,780

GoogleBot? 3,249 3,035 33 181

Desktop 1,045 36 1 1,009

Other 376 118 134 124

Toolbar 274 105 35 134

Image search 114 54 59 1

Maps 111 28 81 1

Earth 98 28 70 1

Translate 31 5 20 6

Mail 11 4 7 1

Wireless 5 3 1 1

FeedFetcher 1 1 1 1

Total 165,047 157,527 3,282 4,236

Top level domains

.com 45,148 43,262 1,278 608

.de 12,673 12,399 166 109

.uk 8,907 8,680 151 76

.fr 7,849 7,661 92 96

.jp 7,649 6,966 64 619

.br 6,278 6,128 72 78

.it 5,967 5,802 97 69

.pl 5,431 5,288 97 46

.ca 5,341 5,202 84 55

.mx 5,286 5,146 60 80

.es 4,730 4,576 78 75

.in 3,110 2,916 67 128

.au 2,846 2,784 43 19

.nl 2,042 1,976 46 21

.ru 1,952 1,818 106 28

.ar 1,896 1,834 29 33

.co 1,859 1,820 18 21

.ve 1,603 1,564 12 27

.tr 1,451 1,380 33 37

.se 1,296 1,269 19 9

.pe 1,267 1,186 15 65

.ph 1,262 1,227 22 14

.be 1,243 1,204 24 14

.cl 1,216 1,183 19 14

.ch 1,173 1,141 20 12

.at 1,143 1,115 15 12

.id 1,081 1,018 51 13

.th 1,030 889 104 37

.fi 992 975 10 7

.ua 942 909 16 17

.pt 843 818 16 9

.hu 789 755 23 10

.hk 746 662 28 56

.ro 734 691 21 22

.vn 677 585 50 42

.il 642 617 10 15

.my 631 597 29 6

.dk 622 603 14 5

.no 575 558 13 4

.sg 531 513 13 5

.gr 530 511 12 7

.ie 494 481 6 6

.ec 484 470 5 10

.bg 473 453 10 10

.cz 470 450 17 4

.nz 447 425 7 15

.tw 400 378 8 14

.hr 397 381 10 6

.do 335 326 3 6

.sk 318 300 12 6

.eg 306 280 6 19

.lt 304 293 7 5

.pk 298 278 7 14

.za 274 263 7 3

.ma 252 233 4 16

.si 209 201 5 3

.sa 208 194 5 9

.ae 186 181 3 1

.kr 178 134 25 20

.cr 173 168 3 2

.uy 170 166 2 2

.pr 166 162 2 2

.lv 147 142 3 2

.bo 146 137 2 7

.ee 143 139 3 1

.gt 131 127 2 2

.ba 127 121 4 2

.sv 102 99 2 1

.py 80 79 1 1

.kz 78 74 2 1

.lk 77 72 3 2

.ke 76 74 1 1

.hn 74 72 1 1

.rs 69 65 2 2

.md 66 64 1 1

.ng 62 59 1 2

.jo 62 59 2 2

.jm 59 57 1 1

.tt 57 56 1 1

.by 51 48 2 1

.ge 48 45 2 1

.lu 46 44 1 1

.is 44 43 1 1

.kw 43 41 1 1

.lb 41 40 1 1

.qa 41 40 1 1

.ps 38 35 1 2

.mt 38 37 1 1

.org 35 4 30 1

.bh 32 31 1 1

.gh 31 30 1 1

.om 27 26 1 1

.mu 27 27 1 1

.az 26 24 1 1

.net 24 3 21 1

.am 23 22 1 1

.ni 23 22 1 1

.ug 22 21 1 1

.cm 22 21 1 1

.cu 22 22 1 1

.pa 21 20 1 1

.np 21 20 1 1

.ly 20 18 1 2

.mn 18 16 1 1

.et 17 16 1 1

.ci 15 14 1 1

.sn 13 13 1 1

.cat 13 13 1 1

.tz 12 11 1 1

.bs 12 12 1 1

.dz 11 9 1 1

.bn 11 11 1 1

.mv 10 10 1

.mz 9 8 1

.fj 8 8 1 1

.bz 8 7 1 1

.zw 8 8 1 1

.bw 7 7 1 1

.mg 7 7 1 1

.kh 7 6 1 1

.gy 7 7 1 1

.kg 6 6 1 1

.gp 6 5 1 1

.rw 5 5 1 1

.dm 5 5 1

.na 5 5 1 1

.uz 5 5 1 1

.bd 4 3 1 1

.cd 4 4 1 1

.cn 4 3 1 1

.ag 4 4 1 1

.vi 4 4 1

.ht 4 4 1 1

.zm 4 4 1

.mk 4 4 1 1

.af 3 3 1 1

.ml 3 3 1

.vc 3 3 1 1

.bf 2 2 1 1

.gi 2 2 1

.la 2 2 1 1

.ga 2 2 1 1

.me 2 2 1 1

.li 2 2 1

.bj 2 2 1

.mw 2 2 1

.dj 2 2 1

.ne 1 1 1

.biz 1 1 1 1

.gl 1 1 1

.ls 1 1 1

.sb 1 1 1 1

.ai 1 1

.vg 1 1 1

.ir 1 1 1

.sl 1 1 1

.us 1 1 1

.bi 1 1 1

.as 1 1 1 1

.tk 1 1 1

.sc 1 1 1

.edu 1 1 1 1

.gm 1 1

.to 1 1 1

.eu 1 1 1

.cc 1 1 1

.tl 1 1 1

.fm 1 1 1

.ws 1 1 1

.ad 1 1

.tg 1 1 1

.sm 1 1 1

.gg 1 1 1

.ki 1 1

.nu 1 1 1

.st 1 1 1

.ck 1 1

.ao 1 1

.tm 1 1

.ms 1 1 1

.pn 1 1

.gs 1 1

.td 1 1

.cg 1 1

.tv 1 1

.gd 1 1

.su 1 1

.nr 1 1

.vu 1 1 1

.ac 1 1 1

.je 1 1

.sh 1 1

.tj 1 1

.cf 1 1

.im 1 1 1

undefined 17,278 14,315 567 2,396

Total 178,434 169,227 3,962 5,240

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,918 3,858 57 4

Web search 14 13 1 1

Translate 1,026 1,026 1 1

Toolbar 1 1

Other 1,218 658 222 339

Maps 1 1 1

GoogleBot 6,915 6,059 399 457

FeedFetcher 304 88 11 204

Desktop 1 1

Total 13,395 11,702 689 1,005

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 5 3 1 1

Web search 159,732 154,110 2,842 2,780

Translate 31 5 20 6

Toolbar 274 105 35 134

Other 376 118 134 124

Maps 111 28 81 1

Mail 11 4 7 1

Image search 114 54 59 1

GoogleBot? 3,249 3,035 33 181

FeedFetcher 1 1 1 1

Earth 98 28 70 1

Desktop 1,045 36 1 1,009

Total 165,047 157,527 3,282 4,236

Top level domains

.ac 1 1 1

.ad 1 1

.ae 186 181 3 1

.af 3 3 1 1

.ag 4 4 1 1

.ai 1 1

.am 23 22 1 1

.ao 1 1

.ar 1,896 1,834 29 33

.as 1 1 1 1

.at 1,143 1,115 15 12

.au 2,846 2,784 43 19

.az 26 24 1 1

.ba 127 121 4 2

.bd 4 3 1 1

.be 1,243 1,204 24 14

.bf 2 2 1 1

.bg 473 453 10 10

.bh 32 31 1 1

.bi 1 1 1

.biz 1 1 1 1

.bj 2 2 1

.bn 11 11 1 1

.bo 146 137 2 7

.br 6,278 6,128 72 78

.bs 12 12 1 1

.bw 7 7 1 1

.by 51 48 2 1

.bz 8 7 1 1

.ca 5,341 5,202 84 55

.cat 13 13 1 1

.cc 1 1 1

.cd 4 4 1 1

.cf 1 1

.cg 1 1

.ch 1,173 1,141 20 12

.ci 15 14 1 1

.ck 1 1

.cl 1,216 1,183 19 14

.cm 22 21 1 1

.cn 4 3 1 1

.co 1,859 1,820 18 21

.com 45,148 43,262 1,278 608

.cr 173 168 3 2

.cu 22 22 1 1

.cz 470 450 17 4

.de 12,673 12,399 166 109

.dj 2 2 1

.dk 622 603 14 5

.dm 5 5 1

.do 335 326 3 6

.dz 11 9 1 1

.ec 484 470 5 10

.edu 1 1 1 1

.ee 143 139 3 1

.eg 306 280 6 19

.es 4,730 4,576 78 75

.et 17 16 1 1

.eu 1 1 1

.fi 992 975 10 7

.fj 8 8 1 1

.fm 1 1 1

.fr 7,849 7,661 92 96

.ga 2 2 1 1

.gd 1 1

.ge 48 45 2 1

.gg 1 1 1

.gh 31 30 1 1

.gi 2 2 1

.gl 1 1 1

.gm 1 1

.gp 6 5 1 1

.gr 530 511 12 7

.gs 1 1

.gt 131 127 2 2

.gy 7 7 1 1

.hk 746 662 28 56

.hn 74 72 1 1

.hr 397 381 10 6

.ht 4 4 1 1

.hu 789 755 23 10

.id 1,081 1,018 51 13

.ie 494 481 6 6

.il 642 617 10 15

.im 1 1 1

.in 3,110 2,916 67 128

.ir 1 1 1

.is 44 43 1 1

.it 5,967 5,802 97 69

.je 1 1

.jm 59 57 1 1

.jo 62 59 2 2

.jp 7,649 6,966 64 619

.ke 76 74 1 1

.kg 6 6 1 1

.kh 7 6 1 1

.ki 1 1

.kr 178 134 25 20

.kw 43 41 1 1

.kz 78 74 2 1

.la 2 2 1 1

.lb 41 40 1 1

.li 2 2 1

.lk 77 72 3 2

.ls 1 1 1

.lt 304 293 7 5

.lu 46 44 1 1

.lv 147 142 3 2

.ly 20 18 1 2

.ma 252 233 4 16

.md 66 64 1 1

.me 2 2 1 1

.mg 7 7 1 1

.mk 4 4 1 1

.ml 3 3 1

.mn 18 16 1 1

.ms 1 1 1

.mt 38 37 1 1

.mu 27 27 1 1

.mv 10 10 1

.mw 2 2 1

.mx 5,286 5,146 60 80

.my 631 597 29 6

.mz 9 8 1

.na 5 5 1 1

.ne 1 1 1

.net 24 3 21 1

.ng 62 59 1 2

.ni 23 22 1 1

.nl 2,042 1,976 46 21

.no 575 558 13 4

.np 21 20 1 1

.nr 1 1

.nu 1 1 1

.nz 447 425 7 15

.om 27 26 1 1

.org 35 4 30 1

.pa 21 20 1 1

.pe 1,267 1,186 15 65

.ph 1,262 1,227 22 14

.pk 298 278 7 14

.pl 5,431 5,288 97 46

.pn 1 1

.pr 166 162 2 2

.ps 38 35 1 2

.pt 843 818 16 9

.py 80 79 1 1

.qa 41 40 1 1

.ro 734 691 21 22

.rs 69 65 2 2

.ru 1,952 1,818 106 28

.rw 5 5 1 1

.sa 208 194 5 9

.sb 1 1 1 1

.sc 1 1 1

.se 1,296 1,269 19 9

.sg 531 513 13 5

.sh 1 1

.si 209 201 5 3

.sk 318 300 12 6

.sl 1 1 1

.sm 1 1 1

.sn 13 13 1 1

.st 1 1 1

.su 1 1

.sv 102 99 2 1

.td 1 1

.tg 1 1 1

.th 1,030 889 104 37

.tj 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1

.to 1 1 1

.tr 1,451 1,380 33 37

.tt 57 56 1 1

.tv 1 1

.tw 400 378 8 14

.tz 12 11 1 1

.ua 942 909 16 17

.ug 22 21 1 1

.uk 8,907 8,680 151 76

.us 1 1 1

.uy 170 166 2 2

.uz 5 5 1 1

.vc 3 3 1 1

.ve 1,603 1,564 12 27

.vg 1 1 1

.vi 4 4 1

.vn 677 585 50 42

.vu 1 1 1

.ws 1 1 1

.za 274 263 7 3

.zm 4 4 1

.zw 8 8 1 1

undefined 17,278 14,315 567 2,396

Total 178,434 169,227 3,962 5,240

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 1,045 - - Y
Earth 98 - - Y
FeedFetcher 304 Y - Y
GoogleBot 6,915 Y - Y
GoogleBot? 3,202 - - Y
GoogleBot? 47 - Y Y
Image search 114 - Y -
Mail 11 - Y -
Maps 111 - Y -
Other 277 - - Y
Other 8 - Y -
Other 92 - Y Y
Other 462 Y - -
Other 753 Y - Y
Other 3 Y Y Y
Toolbar 184 - - Y
Toolbar 58 - Y -
Toolbar 32 - Y Y
Translate 30 - Y -
Translate 796 Y - Y
Translate 230 Y Y Y
Web search 159,732 - Y -
Web search 14 Y Y -
Wireless 5 - - Y
Wireless 3,918 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Tue, Jan 11, 2011 6:32
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.