Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Sep 2011 - 30 Sep 2011

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google
WMF traffic logging service suffered from server capacity problems in August and September 2011.
Absolute traffic counts for September 2011 are approximatly 19% too low.
Data loss only occurred during peak hours. It therefore may have had different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 43.4% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 121,079,000 page views per day, representing 36.6% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 22,265,000 page pages per day, representing 6.7% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 19,089 17,184 746 1,159

Wireless 3,314 3,249 62 3

Other 2,340 942 416 983

Translate 799 796 3

FeedFetcher 358 81 75 203

Web search 14 13 1 1

Image search 1 1

Toolbar 1 1

Desktop 1 1

Maps 1 1

Total 25,914 22,265 1,300 2,351

Requests originating from elsewhere

Service Total Pages Images Other

Web search 132,658 116,399 2,089 14,170

GoogleBot? 5,251 4,516 40 696

Desktop 538 19 519

Other 329 45 73 211

Maps 112 22 89 1

Toolbar 98 32 12 53

Image search 50 28 20 1

Earth 28 14 14 1

Translate 13 3 8 2

Mail 3 1 2 1

Wireless 1 1 1 1

FeedFetcher 1 1 1 1

Total 139,082 121,079 2,348 15,653

Top level domains

.com 41,775 32,204 984 8,588

.de 7,286 6,775 66 444

.uk 6,991 6,064 121 806

.jp 6,425 5,326 23 1,076

.mx 5,944 5,659 82 203

.fr 5,242 4,773 83 385

.br 5,046 4,886 68 92

.it 4,748 4,425 73 250

.ca 3,981 3,565 62 354

.in 3,500 3,264 69 166

.es 3,277 3,040 56 182

.pl 2,907 2,816 51 41

.au 2,775 2,439 45 291

.co 2,335 2,273 28 34

.ru 1,984 1,877 34 73

.ar 1,808 1,751 30 28

.nl 1,558 1,417 36 105

.ph 1,363 1,310 29 23

.pe 1,183 1,140 24 19

.th 1,064 1,007 23 34

.tr 1,057 993 24 40

.se 1,040 921 18 102

.cl 1,017 971 20 26

.id 939 885 28 26

.be 847 806 19 22

.ua 841 817 11 13

.ch 834 745 19 70

.at 706 636 15 55

.fi 654 621 11 23

.my 638 607 11 20

.ve 624 601 8 15

.hk 623 575 9 39

.vn 575 542 9 24

.il 531 481 8 41

.sg 524 454 5 64

.nz 523 490 8 25

.hu 488 465 15 8

.ec 480 465 7 8

.dk 472 426 9 37

.ro 460 433 12 15

.no 459 419 10 29

.pt 450 433 7 10

.gr 421 402 9 10

.do 417 406 3 7

.ie 385 337 5 44

.tw 369 356 5 8

.cz 349 330 12 7

.pk 336 315 8 14

.bg 299 288 6 5

.za 298 268 7 23

.hr 265 253 6 6

.gt 260 251 4 5

.org 243 5 237 1

.sa 230 207 6 18

.eg 222 206 7 9

.sk 195 185 6 4

.lt 189 182 4 2

.ma 186 176 4 7

.ae 181 168 4 10

.cr 164 152 4 8

.uy 155 150 3 2

.sv 154 148 3 4

.kr 154 123 5 26

.pr 152 142 3 7

.bo 150 145 3 3

.si 131 126 4 2

.by 99 96 2 1

.ba 96 92 3 1

.ee 96 93 2 1

.lv 93 90 2 1

.kz 91 88 1 2

.hn 88 84 2 2

.py 81 77 1 3

.lk 80 76 3 2

.tn 73 70 2 1

.ng 65 60 1 4

.jo 63 60 2 1

.rs 60 57 1 1

.md 58 56 1 1

.jm 56 53 1 1

.tt 54 52 1 1

.ni 50 48 1 2

.ke 46 43 1 3

.qa 43 40 1 1

.ge 39 38 1 1

.kw 38 34 1 3

.is 35 34 1 1

.ws 32 1 31 1

.lb 31 29 1 1

.np 31 29 1 1

.lu 30 26 1 4

.az 30 28 1 1

.mu 29 28 1 1

.ps 28 26 1 1

.gh 28 25 1 2

.mt 27 26 1 1

.am 27 26 1 1

.iq 25 23 1 1

.om 24 23 1 1

.bh 23 21 1 1

.net 18 3 15 1

.ug 16 15 1 1

.pa 16 15 1 1

.cm 13 12 1 1

.mn 13 12 1 1

.ly 12 10 1 1

.bn 12 11 1 1

.mv 11 10 1 1

.sn 11 10 1 1

.tz 11 10 1 1

.ci 10 10 1 1

.mz 10 9 1 1

.bs 10 10 1 1

.cu 10 10 1 1

.zw 10 10 1 1

.et 9 8 1 1

.bw 8 7 1 1

.kh 8 7 1 1

.cat 8 8 1 1

.dm 7 7 1 1

.kg 7 7 1 1

.mg 7 6 1 1

.bz 7 7 1 1

.na 6 6 1 1

.dz 6 6 1 1

.gy 6 6 1 1

.zm 6 5 1 1

.uz 5 5 1 1

.vc 5 4 1 1

.gp 5 5 1

.bd 4 3 1 1

.rw 4 4 1 1

.cd 4 3 1 1

.ht 4 3 1

.af 3 2 1 1

.li 3 2 1 1

.ag 3 3 1 1

.vi 3 3 1

.mw 3 2 1 1

.mk 3 3 1 1

.bf 2 2 1 1

.ao 2 2 1

.la 2 2 1 1

.me 2 1 1 1

.fj 2 2 1 1

.ml 2 2 1 1

.dj 2 2 1

.ne 1 1 1 1

.biz 1 1 1

.ls 1 1 1

.tm 1 1 1

.ga 1 1 1 1

.gi 1 1 1 1

.sb 1 1 1

.ai 1 1

.vg 1 1

.ir 1 1 1

.sl 1 1 1

.us 1 1 1

.as 1 1 1 1

.sc 1 1 1

.bj 1 1 1

.gm 1 1 1

.to 1 1 1 1

.so 1 1 1 1

.tl 1 1 1

.fm 1 1 1

.ad 1 1 1 1

.tg 1 1 1

.gg 1 1 1

.ki 1 1

.gl 1 1

.nu 1 1 1

.cy 1 1 1

.st 1 1 1

.ck 1 1

.tc 1 1

.ms 1 1

.pn 1 1

.td 1 1 1

.cg 1 1 1

.tv 1 1 1

.bi 1 1 1

.gd 1 1

.tk 1 1 1

.su 1 1 1

.cn 1 1 1 1

.nr 1 1

.vu 1 1

.edu 1 1 1

.eu 1 1 1

.ac 1 1

.gov 1 1

.je 1 1

.cc 1 1 1

.sh 1 1

.tj 1 1

.cf 1 1

.im 1 1 1

.sm 1 1

undefined 30,988 26,265 919 3,804

Total 164,989 143,344 3,639 18,000

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,314 3,249 62 3

Web search 14 13 1 1

Translate 799 796 3

Toolbar 1 1

Other 2,340 942 416 983

Maps 1 1

Image search 1 1

GoogleBot 19,089 17,184 746 1,159

FeedFetcher 358 81 75 203

Desktop 1 1

Total 25,914 22,265 1,300 2,351

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 1 1 1 1

Web search 132,658 116,399 2,089 14,170

Translate 13 3 8 2

Toolbar 98 32 12 53

Other 329 45 73 211

Maps 112 22 89 1

Mail 3 1 2 1

Image search 50 28 20 1

GoogleBot? 5,251 4,516 40 696

FeedFetcher 1 1 1 1

Earth 28 14 14 1

Desktop 538 19 519

Total 139,082 121,079 2,348 15,653

Top level domains

.ac 1 1

.ad 1 1 1 1

.ae 181 168 4 10

.af 3 2 1 1

.ag 3 3 1 1

.ai 1 1

.am 27 26 1 1

.ao 2 2 1

.ar 1,808 1,751 30 28

.as 1 1 1 1

.at 706 636 15 55

.au 2,775 2,439 45 291

.az 30 28 1 1

.ba 96 92 3 1

.bd 4 3 1 1

.be 847 806 19 22

.bf 2 2 1 1

.bg 299 288 6 5

.bh 23 21 1 1

.bi 1 1 1

.biz 1 1 1

.bj 1 1 1

.bn 12 11 1 1

.bo 150 145 3 3

.br 5,046 4,886 68 92

.bs 10 10 1 1

.bw 8 7 1 1

.by 99 96 2 1

.bz 7 7 1 1

.ca 3,981 3,565 62 354

.cat 8 8 1 1

.cc 1 1 1

.cd 4 3 1 1

.cf 1 1

.cg 1 1 1

.ch 834 745 19 70

.ci 10 10 1 1

.ck 1 1

.cl 1,017 971 20 26

.cm 13 12 1 1

.cn 1 1 1 1

.co 2,335 2,273 28 34

.com 41,775 32,204 984 8,588

.cr 164 152 4 8

.cu 10 10 1 1

.cy 1 1 1

.cz 349 330 12 7

.de 7,286 6,775 66 444

.dj 2 2 1

.dk 472 426 9 37

.dm 7 7 1 1

.do 417 406 3 7

.dz 6 6 1 1

.ec 480 465 7 8

.edu 1 1 1

.ee 96 93 2 1

.eg 222 206 7 9

.es 3,277 3,040 56 182

.et 9 8 1 1

.eu 1 1 1

.fi 654 621 11 23

.fj 2 2 1 1

.fm 1 1 1

.fr 5,242 4,773 83 385

.ga 1 1 1 1

.gd 1 1

.ge 39 38 1 1

.gg 1 1 1

.gh 28 25 1 2

.gi 1 1 1 1

.gl 1 1

.gm 1 1 1

.gov 1 1

.gp 5 5 1

.gr 421 402 9 10

.gt 260 251 4 5

.gy 6 6 1 1

.hk 623 575 9 39

.hn 88 84 2 2

.hr 265 253 6 6

.ht 4 3 1

.hu 488 465 15 8

.id 939 885 28 26

.ie 385 337 5 44

.il 531 481 8 41

.im 1 1 1

.in 3,500 3,264 69 166

.iq 25 23 1 1

.ir 1 1 1

.is 35 34 1 1

.it 4,748 4,425 73 250

.je 1 1

.jm 56 53 1 1

.jo 63 60 2 1

.jp 6,425 5,326 23 1,076

.ke 46 43 1 3

.kg 7 7 1 1

.kh 8 7 1 1

.ki 1 1

.kr 154 123 5 26

.kw 38 34 1 3

.kz 91 88 1 2

.la 2 2 1 1

.lb 31 29 1 1

.li 3 2 1 1

.lk 80 76 3 2

.ls 1 1 1

.lt 189 182 4 2

.lu 30 26 1 4

.lv 93 90 2 1

.ly 12 10 1 1

.ma 186 176 4 7

.md 58 56 1 1

.me 2 1 1 1

.mg 7 6 1 1

.mk 3 3 1 1

.ml 2 2 1 1

.mn 13 12 1 1

.ms 1 1

.mt 27 26 1 1

.mu 29 28 1 1

.mv 11 10 1 1

.mw 3 2 1 1

.mx 5,944 5,659 82 203

.my 638 607 11 20

.mz 10 9 1 1

.na 6 6 1 1

.ne 1 1 1 1

.net 18 3 15 1

.ng 65 60 1 4

.ni 50 48 1 2

.nl 1,558 1,417 36 105

.no 459 419 10 29

.np 31 29 1 1

.nr 1 1

.nu 1 1 1

.nz 523 490 8 25

.om 24 23 1 1

.org 243 5 237 1

.pa 16 15 1 1

.pe 1,183 1,140 24 19

.ph 1,363 1,310 29 23

.pk 336 315 8 14

.pl 2,907 2,816 51 41

.pn 1 1

.pr 152 142 3 7

.ps 28 26 1 1

.pt 450 433 7 10

.py 81 77 1 3

.qa 43 40 1 1

.ro 460 433 12 15

.rs 60 57 1 1

.ru 1,984 1,877 34 73

.rw 4 4 1 1

.sa 230 207 6 18

.sb 1 1 1

.sc 1 1 1

.se 1,040 921 18 102

.sg 524 454 5 64

.sh 1 1

.si 131 126 4 2

.sk 195 185 6 4

.sl 1 1 1

.sm 1 1

.sn 11 10 1 1

.so 1 1 1 1

.st 1 1 1

.su 1 1 1

.sv 154 148 3 4

.tc 1 1

.td 1 1 1

.tg 1 1 1

.th 1,064 1,007 23 34

.tj 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1 1

.tn 73 70 2 1

.to 1 1 1 1

.tr 1,057 993 24 40

.tt 54 52 1 1

.tv 1 1 1

.tw 369 356 5 8

.tz 11 10 1 1

.ua 841 817 11 13

.ug 16 15 1 1

.uk 6,991 6,064 121 806

.us 1 1 1

.uy 155 150 3 2

.uz 5 5 1 1

.vc 5 4 1 1

.ve 624 601 8 15

.vg 1 1

.vi 3 3 1

.vn 575 542 9 24

.vu 1 1

.ws 32 1 31 1

.za 298 268 7 23

.zm 6 5 1 1

.zw 10 10 1 1

undefined 30,988 26,265 919 3,804

Total 164,989 143,344 3,639 18,000

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 538 - - Y
Earth 28 - - Y
FeedFetcher 358 Y - Y
GoogleBot 19,089 Y - Y
GoogleBot? 5,218 - - Y
GoogleBot? 34 - Y Y
Image search 50 - Y -
Mail 3 - Y -
Maps 112 - Y -
Other 216 - - Y
Other 46 - Y -
Other 67 - Y Y
Other 148 Y - -
Other 2,190 Y - Y
Toolbar 66 - - Y
Toolbar 20 - Y -
Toolbar 11 - Y Y
Translate 12 - Y -
Translate 644 Y - Y
Translate 155 Y Y Y
Web search 132,658 - Y -
Web search 14 Y Y -
Wireless 3,314 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Thu, Oct 6, 2011 15:00
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.