Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Feb 2011 - 28 Feb 2011

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 41.8% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 147,353,000 page views per day, representing 35.7% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 25,347,000 page pages per day, representing 6.1% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 20,945 19,123 493 1,329

Wireless 4,064 4,002 58 3

Other 1,759 1,052 238 469

Translate 1,063 1,063 1 1

FeedFetcher 320 95 17 208

Web search 12 11 1 1

Toolbar 1 1

Desktop 1 1 1

Total 28,164 25,347 806 2,009

Requests originating from elsewhere

Service Total Pages Images Other

Web search 146,008 141,443 2,542 2,023

GoogleBot? 5,899 5,548 44 306

Desktop 1,018 60 1 958

Other 406 115 133 158

Toolbar 239 85 29 125

Maps 112 28 84 1

Earth 98 31 67 1

Image search 71 34 37 1

Translate 25 5 17 4

Mail 8 2 6 1

Wireless 1 1 1 1

FeedFetcher 1 1 1

Total 153,886 147,353 2,960 3,576

Top level domains

.com 50,366 48,634 1,239 492

.de 9,770 9,595 114 62

.uk 8,624 8,434 137 53

.jp 7,349 6,639 49 661

.fr 6,766 6,623 80 63

.it 5,895 5,740 108 47

.ca 4,881 4,766 79 36

.br 4,429 4,336 52 41

.es 4,364 4,239 77 48

.pl 4,045 3,941 82 23

.mx 3,060 2,999 29 32

.in 2,965 2,786 72 108

.au 2,496 2,442 41 12

.ru 2,099 1,988 95 15

.nl 2,006 1,944 50 12

.tr 1,394 1,342 29 22

.co 1,365 1,344 11 11

.se 1,240 1,211 25 4

.be 1,180 1,145 26 10

.ph 1,101 1,064 27 10

.ch 1,014 987 20 7

.th 984 878 78 28

.ua 921 899 13 9

.at 921 897 17 7

.ar 883 861 12 10

.fi 789 771 14 3

.pt 770 749 15 7

.id 770 717 45 9

.ve 721 705 6 9

.ro 671 637 19 15

.hu 670 639 26 5

.il 617 594 12 10

.hk 604 561 17 25

.my 586 554 27 5

.no 528 515 11 2

.dk 519 505 12 3

.vn 518 454 37 27

.gr 509 492 12 5

.sg 474 460 11 3

.ie 472 463 6 3

.cz 451 430 18 3

.bg 441 421 11 8

.cl 429 419 6 4

.nz 382 374 7 1

.pe 365 350 4 11

.hr 356 342 11 4

.tw 331 316 5 10

.za 328 313 10 5

.pk 306 284 11 11

.sk 284 270 11 3

.lt 260 250 7 3

.eg 228 211 8 9

.ma 221 207 5 10

.sa 221 211 5 5

.ae 191 186 4 1

.gt 187 182 2 3

.si 181 174 6 2

.do 164 161 1 2

.ec 148 145 2 1

.pr 136 133 2 1

.kr 129 108 12 9

.ba 127 121 5 2

.lv 126 122 3 1

.ee 117 114 3 1

.sv 101 98 2 1

.by 89 87 1 1

.kz 83 81 1 1

.cr 83 81 1 1

.bo 76 73 1 2

.lk 72 66 4 2

.rs 72 68 2 1

.ng 69 66 1 2

.jo 68 65 2 2

.md 67 65 1 1

.uy 62 61 1 1

.ke 59 57 1 1

.ge 50 47 3 1

.tt 49 48 1 1

.jm 48 47 1 1

.hn 47 46 1 1

.kw 41 39 1 1

.org 40 2 37 1

.qa 40 38 1 1

.lu 39 38 1 1

.lb 38 36 1 1

.is 35 34 1 1

.az 34 31 2 1

.mt 33 32 1 1

.py 33 32 1 1

.ps 29 27 1 1

.am 28 26 1 1

.om 26 25 1 1

.gh 26 25 1 1

.mu 26 25 1 1

.bh 24 24 1

.net 24 3 20 1

.ni 23 22 1 1

.np 18 17 1 1

.cm 17 16 1 1

.ly 16 15 1 1

.ug 14 14 1 1

.ci 14 13 1 1

.tz 14 13 1 1

.cat 14 13 1 1

.sn 13 13 1 1

.mn 11 10 1 1

.et 11 11 1 1

.mz 10 10 1 1

.bn 10 9 1 1

.pa 9 9 1 1

.bs 9 9 1

.mv 8 7 1

.dz 8 7 1 1

.bw 8 8 1 1

.zw 8 8 1 1

.kg 7 7 1 1

.mg 7 6 1 1

.kh 7 6 1 1

.uz 7 7 1 1

.cu 7 7 1 1

.gp 7 7 1 1

.na 6 5 1 1

.fj 6 6 1 1

.bz 6 6 1 1

.ht 6 6 1 1

.vc 6 5 1 1

.rw 4 4 1

.dm 4 4 1

.cd 4 4 1 1

.ag 4 4 1

.vi 4 4 1

.gy 4 4 1

.zm 4 4 1 1

.bd 3 3 1 1

.bf 3 3 1

.mw 3 2 1

.eu 3 1 3

.mk 3 3 1 1

.ao 2 2 1

.af 2 2 1 1

.gi 2 2 1

.ga 2 2

.me 2 2 1 1

.li 2 2 1

.us 2 1 1

.ml 2 2 1 1

.cc 2 1 2

.gg 1 1 1

.ne 1 1

.biz 1 1 1 1

.gl 1 1

.ls 1 1

.la 1 1 1 1

.ai 1 1

.vg 1 1

.ir 1 1 1

.cg 1 1 1

.sl 1 1

.as 1 1

.sc 1 1

.bj 1 1 1

.edu 1 1 1 1

.gm 1 1 1 1

.tl 1 1 1

.fm 1 1 1

.ws 1 1 1

.ad 1 1 1

.tg 1 1 1 1

.sm 1 1 1

.dj 1 1 1

.ki 1 1 1

.nu 1 1 1

.st 1 1 1

.hm 1 1

.tm 1 1

.ms 1 1 1

.pn 1 1

.sb 1 1

.td 1 1

.tv 1 1 1

.bi 1 1

.gd 1 1

.tk 1 1 1

.cn 1 1 1

.nr 1 1

.vu 1 1

.nf 1 1

.to 1 1

.gov 1 1

.je 1 1 1

.sh 1 1

.cf 1 1

.tj 1 1 1 1

.im 1 1 1

.pro 1 1

undefined 34,600 30,444 665 3,491

Total 182,047 172,696 3,759 5,581

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 4,064 4,002 58 3

Web search 12 11 1 1

Translate 1,063 1,063 1 1

Toolbar 1 1

Other 1,759 1,052 238 469

GoogleBot 20,945 19,123 493 1,329

FeedFetcher 320 95 17 208

Desktop 1 1 1

Total 28,164 25,347 806 2,009

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 1 1 1 1

Web search 146,008 141,443 2,542 2,023

Translate 25 5 17 4

Toolbar 239 85 29 125

Other 406 115 133 158

Maps 112 28 84 1

Mail 8 2 6 1

Image search 71 34 37 1

GoogleBot? 5,899 5,548 44 306

FeedFetcher 1 1 1

Earth 98 31 67 1

Desktop 1,018 60 1 958

Total 153,886 147,353 2,960 3,576

Top level domains

.ad 1 1 1

.ae 191 186 4 1

.af 2 2 1 1

.ag 4 4 1

.ai 1 1

.am 28 26 1 1

.ao 2 2 1

.ar 883 861 12 10

.as 1 1

.at 921 897 17 7

.au 2,496 2,442 41 12

.az 34 31 2 1

.ba 127 121 5 2

.bd 3 3 1 1

.be 1,180 1,145 26 10

.bf 3 3 1

.bg 441 421 11 8

.bh 24 24 1

.bi 1 1

.biz 1 1 1 1

.bj 1 1 1

.bn 10 9 1 1

.bo 76 73 1 2

.br 4,429 4,336 52 41

.bs 9 9 1

.bw 8 8 1 1

.by 89 87 1 1

.bz 6 6 1 1

.ca 4,881 4,766 79 36

.cat 14 13 1 1

.cc 2 1 2

.cd 4 4 1 1

.cf 1 1

.cg 1 1 1

.ch 1,014 987 20 7

.ci 14 13 1 1

.cl 429 419 6 4

.cm 17 16 1 1

.cn 1 1 1

.co 1,365 1,344 11 11

.com 50,366 48,634 1,239 492

.cr 83 81 1 1

.cu 7 7 1 1

.cz 451 430 18 3

.de 9,770 9,595 114 62

.dj 1 1 1

.dk 519 505 12 3

.dm 4 4 1

.do 164 161 1 2

.dz 8 7 1 1

.ec 148 145 2 1

.edu 1 1 1 1

.ee 117 114 3 1

.eg 228 211 8 9

.es 4,364 4,239 77 48

.et 11 11 1 1

.eu 3 1 3

.fi 789 771 14 3

.fj 6 6 1 1

.fm 1 1 1

.fr 6,766 6,623 80 63

.ga 2 2

.gd 1 1

.ge 50 47 3 1

.gg 1 1 1

.gh 26 25 1 1

.gi 2 2 1

.gl 1 1

.gm 1 1 1 1

.gov 1 1

.gp 7 7 1 1

.gr 509 492 12 5

.gt 187 182 2 3

.gy 4 4 1

.hk 604 561 17 25

.hm 1 1

.hn 47 46 1 1

.hr 356 342 11 4

.ht 6 6 1 1

.hu 670 639 26 5

.id 770 717 45 9

.ie 472 463 6 3

.il 617 594 12 10

.im 1 1 1

.in 2,965 2,786 72 108

.ir 1 1 1

.is 35 34 1 1

.it 5,895 5,740 108 47

.je 1 1 1

.jm 48 47 1 1

.jo 68 65 2 2

.jp 7,349 6,639 49 661

.ke 59 57 1 1

.kg 7 7 1 1

.kh 7 6 1 1

.ki 1 1 1

.kr 129 108 12 9

.kw 41 39 1 1

.kz 83 81 1 1

.la 1 1 1 1

.lb 38 36 1 1

.li 2 2 1

.lk 72 66 4 2

.ls 1 1

.lt 260 250 7 3

.lu 39 38 1 1

.lv 126 122 3 1

.ly 16 15 1 1

.ma 221 207 5 10

.md 67 65 1 1

.me 2 2 1 1

.mg 7 6 1 1

.mk 3 3 1 1

.ml 2 2 1 1

.mn 11 10 1 1

.ms 1 1 1

.mt 33 32 1 1

.mu 26 25 1 1

.mv 8 7 1

.mw 3 2 1

.mx 3,060 2,999 29 32

.my 586 554 27 5

.mz 10 10 1 1

.na 6 5 1 1

.ne 1 1

.net 24 3 20 1

.nf 1 1

.ng 69 66 1 2

.ni 23 22 1 1

.nl 2,006 1,944 50 12

.no 528 515 11 2

.np 18 17 1 1

.nr 1 1

.nu 1 1 1

.nz 382 374 7 1

.om 26 25 1 1

.org 40 2 37 1

.pa 9 9 1 1

.pe 365 350 4 11

.ph 1,101 1,064 27 10

.pk 306 284 11 11

.pl 4,045 3,941 82 23

.pn 1 1

.pr 136 133 2 1

.pro 1 1

.ps 29 27 1 1

.pt 770 749 15 7

.py 33 32 1 1

.qa 40 38 1 1

.ro 671 637 19 15

.rs 72 68 2 1

.ru 2,099 1,988 95 15

.rw 4 4 1

.sa 221 211 5 5

.sb 1 1

.sc 1 1

.se 1,240 1,211 25 4

.sg 474 460 11 3

.sh 1 1

.si 181 174 6 2

.sk 284 270 11 3

.sl 1 1

.sm 1 1 1

.sn 13 13 1 1

.st 1 1 1

.sv 101 98 2 1

.td 1 1

.tg 1 1 1 1

.th 984 878 78 28

.tj 1 1 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1

.to 1 1

.tr 1,394 1,342 29 22

.tt 49 48 1 1

.tv 1 1 1

.tw 331 316 5 10

.tz 14 13 1 1

.ua 921 899 13 9

.ug 14 14 1 1

.uk 8,624 8,434 137 53

.us 2 1 1

.uy 62 61 1 1

.uz 7 7 1 1

.vc 6 5 1 1

.ve 721 705 6 9

.vg 1 1

.vi 4 4 1

.vn 518 454 37 27

.vu 1 1

.ws 1 1 1

.za 328 313 10 5

.zm 4 4 1 1

.zw 8 8 1 1

undefined 34,600 30,444 665 3,491

Total 182,047 172,696 3,759 5,581

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 1,018 - - Y
Earth 98 - - Y
FeedFetcher 320 Y - Y
GoogleBot 20,945 Y - Y
GoogleBot? 5,840 - - Y
GoogleBot? 58 - Y Y
Image search 71 - Y -
Mail 8 - Y -
Maps 112 - Y -
Other 286 - - Y
Other 13 - Y -
Other 108 - Y Y
Other 192 Y - -
Other 1,565 Y - Y
Toolbar 155 - - Y
Toolbar 54 - Y -
Toolbar 29 - Y Y
Translate 24 - Y -
Translate 845 Y - Y
Translate 218 Y Y Y
Web search 146,008 - Y -
Web search 12 Y Y -
Wireless 4,063 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Sat, Mar 5, 2011 14:55
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.