Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Aug 2011 - 31 Aug 2011

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google
WMF traffic logging service suffered from server capacity problems in August and September 2011.
Absolute traffic counts for August 2011 are approximatly 6% too low.
Data loss only occurred during peak hours. It therefore may have had different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 37.7% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 123,387,000 page views per day, representing 31.1% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 26,059,000 page pages per day, representing 6.6% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 22,640 20,092 816 1,732

Wireless 3,872 3,803 66 3

Other 2,742 1,173 492 1,077

Translate 891 889 2

FeedFetcher 380 84 76 220

Web search 19 18 1 1

Image search 1 1

Desktop 1 1

Maps 1 1

Total 30,544 26,059 1,451 3,034

Requests originating from elsewhere

Service Total Pages Images Other

Web search 131,691 116,594 2,095 13,002

GoogleBot? 7,189 6,436 26 727

Desktop 595 19 1 576

Other 486 215 80 190

Toolbar 110 37 12 61

Maps 110 30 78 2

Image search 52 31 20 1

Earth 41 19 22 1

Translate 12 3 7 1

Mail 5 2 3 1

Wireless 1 1 1 1

FeedFetcher 1 1 1

Total 140,293 123,387 2,344 14,560

Top level domains

.com 41,141 32,489 1,004 7,647

.de 7,884 7,413 67 405

.jp 7,174 6,040 28 1,106

.uk 7,022 6,143 95 785

.br 5,334 5,172 76 85

.fr 4,986 4,509 87 390

.mx 4,567 4,343 58 165

.it 4,406 4,084 74 248

.ca 3,843 3,461 52 330

.in 3,752 3,523 84 145

.au 3,149 2,828 56 265

.es 3,011 2,785 54 172

.pl 2,676 2,587 54 34

.co 2,632 2,565 34 33

.ar 2,023 1,957 35 30

.ru 1,709 1,624 35 51

.nl 1,538 1,407 33 99

.ph 1,423 1,367 34 22

.tr 1,136 1,070 27 39

.pe 1,133 1,092 24 17

.cl 1,121 1,073 22 26

.th 1,098 1,044 23 30

.se 1,038 919 18 101

.be 953 909 20 23

.ch 892 804 19 69

.id 764 716 28 20

.at 730 663 16 51

.fi 716 683 12 21

.ua 714 694 11 9

.hk 691 622 13 56

.my 652 620 12 20

.ve 563 543 7 13

.il 561 515 9 37

.sg 553 486 6 61

.vn 541 509 8 24

.nz 516 486 8 21

.pt 512 492 9 11

.hu 511 486 17 8

.ro 468 442 14 12

.dk 452 411 9 32

.ie 427 377 5 45

.tw 418 399 6 12

.no 411 373 9 30

.gr 402 383 8 10

.za 389 353 9 27

.pk 329 305 10 14

.ec 327 314 5 8

.bg 327 316 7 4

.cz 326 308 12 6

.org 288 5 283 1

.gt 285 276 4 5

.hr 247 236 6 5

.do 224 217 2 5

.eg 217 201 7 8

.sa 207 185 4 17

.uy 184 179 3 2

.sk 180 171 6 3

.cr 175 163 4 9

.bo 168 160 4 3

.ma 165 157 4 4

.lt 159 153 4 2

.sv 157 151 2 3

.ae 157 147 3 7

.kr 146 118 5 23

.pr 133 124 2 7

.si 129 124 3 2

.ba 95 91 3 1

.py 92 88 1 3

.ee 92 89 2 1

.by 91 89 2 1

.hn 90 86 2 2

.lv 84 81 2 1

.lk 83 78 3 2

.ng 70 65 1 4

.tn 68 65 2 1

.kz 68 65 1 2

.ni 56 54 1 1

.rs 52 49 1 1

.ke 51 47 1 3

.jo 50 48 1 1

.md 44 43 1 1

.tt 43 42 1 1

.qa 41 39 1 1

.kw 38 33 1 4

.ws 36 1 35 1

.jm 35 34 1 1

.np 33 32 1 1

.ge 32 30 2 1

.mt 31 30 1 1

.lu 31 28 1 3

.az 31 29 1 1

.lb 31 28 1 2

.is 30 30 1 1

.mu 29 28 1 1

.gh 26 24 1 1

.iq 25 23 1 1

.am 22 20 1 1

.ps 21 19 1 1

.bh 21 20 1 1

.pa 21 20 1 1

.net 21 3 18 1

.om 19 18 1 1

.ug 15 14 1 1

.tz 14 12 1 1

.cm 14 13 1 1

.mz 13 12 1 1

.ci 12 11 1 1

.sn 12 11 1 1

.et 12 11 1 1

.bn 12 11 1 1

.mv 11 10 1 1

.kh 10 9 1 1

.mn 9 8 1 1

.zw 9 8 1 1

.bw 8 8 1 1

.bs 8 7 1 1

.gp 8 8 1 1

.na 7 6 1 1

.mg 7 7 1 1

.cat 7 6 1 1

.cu 7 7 1 1

.uz 6 5 1 1

.rw 5 4 1

.dm 5 5 1 1

.ly 5 4 1 1

.kg 5 4 1 1

.zm 5 5 1 1

.dz 4 4 1 1

.bz 4 4 1 1

.ht 4 4 1 1

.vc 4 4 1

.af 3 2 1 1

.li 3 2 1 1

.cd 3 3 1 1

.ag 3 3 1 1

.mw 3 3 1 1

.gy 3 3 1 1

.bd 2 2 1 1

.bf 2 2 1

.ao 2 2 1

.la 2 2 1 1

.vi 2 2 1

.ml 2 2 1

.bj 2 2 1 1

.mk 2 2 1

.biz 1 1 1 1

.ls 1 1

.ga 1 1 1 1

.gi 1 1 1 1

.ai 1 1

.me 1 1 1

.vg 1 1 1

.fj 1 1 1

.ir 1 1 1

.sl 1 1 1 1

.us 1 1 1

.tv 1 1 1

.bi 1 1

.as 1 1 1 1

.sc 1 1 1 1

.gm 1 1 1

.eu 1 1 1

.so 1 1 1 1

.tl 1 1 1

.fm 1 1 1

.ad 1 1

.tg 1 1 1

.dj 1 1 1

.gg 1 1 1

.ne 1 1 1 1

.ki 1 1 1

.gl 1 1 1

.nu 1 1 1

.cy 1 1

.st 1 1 1

.ck 1 1

.tm 1 1 1

.ms 1 1 1

.pn 1 1

.sb 1 1 1

.gs 1 1

.td 1 1

.cg 1 1

.tk 1 1 1

.su 1 1

.cn 1 1 1

.nr 1 1

.vu 1 1 1 1

.edu 1 1 1 1

.to 1 1

.gov 1 1

.je 1 1

.cc 1 1 1

.sh 1 1

.tj 1 1 1

.cf 1 1

.im 1 1 1

.mil 1 1

.sm 1 1

undefined 37,674 32,130 984 4,560

Total 170,838 149,441 3,787 17,589

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,872 3,803 66 3

Web search 19 18 1 1

Translate 891 889 2

Other 2,742 1,173 492 1,077

Maps 1 1

Image search 1 1

GoogleBot 22,640 20,092 816 1,732

FeedFetcher 380 84 76 220

Desktop 1 1

Total 30,544 26,059 1,451 3,034

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 1 1 1 1

Web search 131,691 116,594 2,095 13,002

Translate 12 3 7 1

Toolbar 110 37 12 61

Other 486 215 80 190

Maps 110 30 78 2

Mail 5 2 3 1

Image search 52 31 20 1

GoogleBot? 7,189 6,436 26 727

FeedFetcher 1 1 1

Earth 41 19 22 1

Desktop 595 19 1 576

Total 140,293 123,387 2,344 14,560

Top level domains

.ad 1 1

.ae 157 147 3 7

.af 3 2 1 1

.ag 3 3 1 1

.ai 1 1

.am 22 20 1 1

.ao 2 2 1

.ar 2,023 1,957 35 30

.as 1 1 1 1

.at 730 663 16 51

.au 3,149 2,828 56 265

.az 31 29 1 1

.ba 95 91 3 1

.bd 2 2 1 1

.be 953 909 20 23

.bf 2 2 1

.bg 327 316 7 4

.bh 21 20 1 1

.bi 1 1

.biz 1 1 1 1

.bj 2 2 1 1

.bn 12 11 1 1

.bo 168 160 4 3

.br 5,334 5,172 76 85

.bs 8 7 1 1

.bw 8 8 1 1

.by 91 89 2 1

.bz 4 4 1 1

.ca 3,843 3,461 52 330

.cat 7 6 1 1

.cc 1 1 1

.cd 3 3 1 1

.cf 1 1

.cg 1 1

.ch 892 804 19 69

.ci 12 11 1 1

.ck 1 1

.cl 1,121 1,073 22 26

.cm 14 13 1 1

.cn 1 1 1

.co 2,632 2,565 34 33

.com 41,141 32,489 1,004 7,647

.cr 175 163 4 9

.cu 7 7 1 1

.cy 1 1

.cz 326 308 12 6

.de 7,884 7,413 67 405

.dj 1 1 1

.dk 452 411 9 32

.dm 5 5 1 1

.do 224 217 2 5

.dz 4 4 1 1

.ec 327 314 5 8

.edu 1 1 1 1

.ee 92 89 2 1

.eg 217 201 7 8

.es 3,011 2,785 54 172

.et 12 11 1 1

.eu 1 1 1

.fi 716 683 12 21

.fj 1 1 1

.fm 1 1 1

.fr 4,986 4,509 87 390

.ga 1 1 1 1

.ge 32 30 2 1

.gg 1 1 1

.gh 26 24 1 1

.gi 1 1 1 1

.gl 1 1 1

.gm 1 1 1

.gov 1 1

.gp 8 8 1 1

.gr 402 383 8 10

.gs 1 1

.gt 285 276 4 5

.gy 3 3 1 1

.hk 691 622 13 56

.hn 90 86 2 2

.hr 247 236 6 5

.ht 4 4 1 1

.hu 511 486 17 8

.id 764 716 28 20

.ie 427 377 5 45

.il 561 515 9 37

.im 1 1 1

.in 3,752 3,523 84 145

.iq 25 23 1 1

.ir 1 1 1

.is 30 30 1 1

.it 4,406 4,084 74 248

.je 1 1

.jm 35 34 1 1

.jo 50 48 1 1

.jp 7,174 6,040 28 1,106

.ke 51 47 1 3

.kg 5 4 1 1

.kh 10 9 1 1

.ki 1 1 1

.kr 146 118 5 23

.kw 38 33 1 4

.kz 68 65 1 2

.la 2 2 1 1

.lb 31 28 1 2

.li 3 2 1 1

.lk 83 78 3 2

.ls 1 1

.lt 159 153 4 2

.lu 31 28 1 3

.lv 84 81 2 1

.ly 5 4 1 1

.ma 165 157 4 4

.md 44 43 1 1

.me 1 1 1

.mg 7 7 1 1

.mil 1 1

.mk 2 2 1

.ml 2 2 1

.mn 9 8 1 1

.ms 1 1 1

.mt 31 30 1 1

.mu 29 28 1 1

.mv 11 10 1 1

.mw 3 3 1 1

.mx 4,567 4,343 58 165

.my 652 620 12 20

.mz 13 12 1 1

.na 7 6 1 1

.ne 1 1 1 1

.net 21 3 18 1

.ng 70 65 1 4

.ni 56 54 1 1

.nl 1,538 1,407 33 99

.no 411 373 9 30

.np 33 32 1 1

.nr 1 1

.nu 1 1 1

.nz 516 486 8 21

.om 19 18 1 1

.org 288 5 283 1

.pa 21 20 1 1

.pe 1,133 1,092 24 17

.ph 1,423 1,367 34 22

.pk 329 305 10 14

.pl 2,676 2,587 54 34

.pn 1 1

.pr 133 124 2 7

.ps 21 19 1 1

.pt 512 492 9 11

.py 92 88 1 3

.qa 41 39 1 1

.ro 468 442 14 12

.rs 52 49 1 1

.ru 1,709 1,624 35 51

.rw 5 4 1

.sa 207 185 4 17

.sb 1 1 1

.sc 1 1 1 1

.se 1,038 919 18 101

.sg 553 486 6 61

.sh 1 1

.si 129 124 3 2

.sk 180 171 6 3

.sl 1 1 1 1

.sm 1 1

.sn 12 11 1 1

.so 1 1 1 1

.st 1 1 1

.su 1 1

.sv 157 151 2 3

.td 1 1

.tg 1 1 1

.th 1,098 1,044 23 30

.tj 1 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1 1

.tn 68 65 2 1

.to 1 1

.tr 1,136 1,070 27 39

.tt 43 42 1 1

.tv 1 1 1

.tw 418 399 6 12

.tz 14 12 1 1

.ua 714 694 11 9

.ug 15 14 1 1

.uk 7,022 6,143 95 785

.us 1 1 1

.uy 184 179 3 2

.uz 6 5 1 1

.vc 4 4 1

.ve 563 543 7 13

.vg 1 1 1

.vi 2 2 1

.vn 541 509 8 24

.vu 1 1 1 1

.ws 36 1 35 1

.za 389 353 9 27

.zm 5 5 1 1

.zw 9 8 1 1

undefined 37,674 32,130 984 4,560

Total 170,838 149,441 3,787 17,589

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 595 - - Y
Earth 41 - - Y
FeedFetcher 380 Y - Y
GoogleBot 22,640 Y - Y
GoogleBot? 7,138 - - Y
GoogleBot? 51 - Y Y
Image search 52 - Y -
Mail 5 - Y -
Maps 110 - Y -
Other 394 - - Y
Other 15 - Y -
Other 76 - Y Y
Other 156 Y - -
Other 2,585 Y - Y
Toolbar 76 - - Y
Toolbar 22 - Y -
Toolbar 13 - Y Y
Translate 10 - Y -
Translate 737 Y - Y
Translate 154 Y Y Y
Web search 131,691 - Y -
Web search 19 Y Y -
Wireless 3,872 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Thu, Oct 6, 2011 14:54
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.