Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Dec 2010 - 31 Dec 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 42.2% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 146,453,000 page views per day, representing 38.8% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 12,947,000 page pages per day, representing 3.4% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 8,920 7,696 463 761

Wireless 3,638 3,575 60 3

Other 1,294 663 236 395

Translate 913 913 1 1

FeedFetcher 323 86 18 218

Web search 16 14 1 1

Toolbar 1 1

Desktop 1 1

Total 15,104 12,947 777 1,378

Requests originating from elsewhere

Service Total Pages Images Other

Web search 146,421 141,573 2,525 2,323

GoogleBot? 4,851 4,561 35 255

Desktop 954 34 1 920

Other 328 94 132 102

Toolbar 234 84 28 121

Maps 108 27 80 1

Image search 93 44 48 1

Earth 90 25 65 1

Translate 26 4 17 5

Mail 9 3 7 1

Wireless 4 3 1 1

FeedFetcher 2 1 1

Total 153,120 146,453 2,938 3,729

Top level domains

.com 45,594 43,813 1,248 534

.de 11,208 10,987 147 74

.uk 8,283 8,118 108 57

.jp 7,350 6,702 51 598

.fr 7,248 7,086 84 78

.it 5,599 5,463 86 50

.ca 4,942 4,831 65 46

.pl 4,866 4,742 89 36

.br 4,475 4,379 52 44

.es 3,974 3,865 61 49

.mx 3,485 3,402 41 42

.in 3,451 3,257 70 123

.ru 2,301 2,146 118 36

.au 2,183 2,144 26 13

.nl 1,952 1,897 40 15

.tr 1,802 1,712 46 43

.ar 1,375 1,340 18 17

.se 1,249 1,222 20 6

.be 1,192 1,160 21 10

.ch 1,080 1,052 19 9

.ph 1,075 1,049 17 9

.at 1,015 990 16 8

.ua 1,002 961 23 18

.fi 931 914 13 4

.cl 916 895 12 9

.id 907 856 42 9

.th 904 794 82 28

.co 826 810 8 8

.hu 748 718 23 8

.pe 735 695 8 32

.pt 719 701 13 5

.hk 716 644 25 47

.il 671 647 10 14

.my 670 640 25 5

.ro 663 625 20 17

.ve 661 645 7 9

.vn 628 550 44 35

.sg 530 517 9 5

.gr 528 510 11 6

.no 528 516 10 2

.dk 513 500 9 4

.ie 479 471 5 3

.cz 463 445 15 4

.bg 456 438 10 9

.tw 402 383 7 12

.hr 380 365 10 5

.nz 375 352 5 18

.pk 350 328 9 13

.eg 338 311 8 20

.ec 320 312 3 5

.ma 303 282 4 17

.sk 289 275 11 4

.sa 265 251 5 9

.lt 262 252 8 3

.do 218 213 2 3

.si 199 193 5 2

.za 195 189 4 2

.ae 176 173 3 1

.kr 145 118 16 12

.lv 144 140 3 1

.ba 137 130 4 2

.ee 133 130 2 1

.pr 132 129 2 1

.cr 99 96 1 1

.uy 95 92 2 1

.kz 88 83 3 2

.jo 82 79 2 2

.lk 81 76 3 2

.by 75 72 2 2

.rs 70 65 3 2

.md 70 69 1 1

.sv 63 61 1 1

.bo 59 56 1 2

.gt 59 57 1 1

.ng 57 55 1 2

.ke 53 51 1 1

.kw 52 50 1 1

.ge 49 44 4 1

.py 48 47 1 1

.tt 47 46 1 1

.qa 45 43 1 1

.ps 42 39 1 2

.jm 42 41 1 1

.lb 41 40 1 1

.is 37 37 1 1

.lu 37 37 1 1

.org 36 3 31 1

.mt 35 34 1 1

.az 34 30 2 2

.hn 32 31 1 1

.bh 31 30 1

.net 27 4 23 1

.om 26 25 1 1

.gh 26 26 1 1

.am 25 23 1 1

.ly 24 22 1 2

.np 21 20 1 1

.mu 21 21 1 1

.et 20 19 1 1

.cm 19 19 1 1

.tz 16 15 1 1

.mn 16 15 1 1

.sn 15 14 1 1

.ug 14 13 1 1

.ci 14 13 1 1

.ni 14 13 1 1

.cat 13 12 1 1

.mv 11 11 1 1

.bs 11 11 1 1

.cu 11 11 1 1

.bn 11 11 1 1

.pa 10 10 1 1

.kg 9 8 1 1

.dz 8 7 1 1

.kh 8 7 1 1

.mg 7 6 1 1

.uz 7 6 1 1

.fj 6 6 1 1

.bz 6 6 1 1

.zw 6 5 1 1

.cd 5 5 1 1

.mz 5 5 1 1

.bd 4 4 1 1

.cn 4 3 1 1

.ht 4 4 1

.gy 4 4 1

.zm 4 3 1 1

.mk 4 3 1 1

.vc 4 3 1 1

.gp 4 4 1 1

.rw 3 3 1 1

.dm 3 3 1

.af 3 3 1 1

.na 3 3 1 1

.bw 3 3 1 1

.ag 3 3

.vi 3 3 1

.biz 2 1 1 1

.bf 2 2 1 1

.la 2 2 1 1

.gi 2 2

.ga 2 2 1 1

.me 2 2 1 1

.li 2 2

.us 2 1 1

.ml 2 2 1 1

.mw 2 2 1

.dj 2 2 1 1

.ne 1 1 1

.gl 1 1 1

.ao 1 1 1 1

.ls 1 1

.tm 1 1 1 1

.ai 1 1 1

.vg 1 1 1 1

.ir 1 1 1

.sl 1 1 1

.tv 1 1 1

.bi 1 1 1

.as 1 1 1

.sc 1 1 1 1

.bj 1 1 1 1

.edu 1 1 1 1

.gm 1 1 1

.eu 1 1 1

.je 1 1

.cc 1 1 1 1

.tl 1 1 1 1

.fm 1 1 1

.ws 1 1 1 1

.ad 1 1 1

.tg 1 1 1

.sm 1 1

.gg 1 1 1

.ki 1 1

.nu 1 1 1

.st 1 1

.ck 1 1

.ms 1 1 1

.pn 1 1

.sb 1 1

.td 1 1

.cg 1 1

.tk 1 1 1 1

.su 1 1

.cx 1 1

.nr 1 1

.vu 1 1

.to 1 1 1

.ac 1 1

.sh 1 1

.cf 1 1

.tj 1 1 1 1

.im 1 1

.pro 1 1

.cv 1 1

undefined 20,476 17,088 635 2,753

Total 168,223 159,398 3,712 5,103

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,638 3,575 60 3

Web search 16 14 1 1

Translate 913 913 1 1

Toolbar 1 1

Other 1,294 663 236 395

GoogleBot 8,920 7,696 463 761

FeedFetcher 323 86 18 218

Desktop 1 1

Total 15,104 12,947 777 1,378

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 4 3 1 1

Web search 146,421 141,573 2,525 2,323

Translate 26 4 17 5

Toolbar 234 84 28 121

Other 328 94 132 102

Maps 108 27 80 1

Mail 9 3 7 1

Image search 93 44 48 1

GoogleBot? 4,851 4,561 35 255

FeedFetcher 2 1 1

Earth 90 25 65 1

Desktop 954 34 1 920

Total 153,120 146,453 2,938 3,729

Top level domains

.ac 1 1

.ad 1 1 1

.ae 176 173 3 1

.af 3 3 1 1

.ag 3 3

.ai 1 1 1

.am 25 23 1 1

.ao 1 1 1 1

.ar 1,375 1,340 18 17

.as 1 1 1

.at 1,015 990 16 8

.au 2,183 2,144 26 13

.az 34 30 2 2

.ba 137 130 4 2

.bd 4 4 1 1

.be 1,192 1,160 21 10

.bf 2 2 1 1

.bg 456 438 10 9

.bh 31 30 1

.bi 1 1 1

.biz 2 1 1 1

.bj 1 1 1 1

.bn 11 11 1 1

.bo 59 56 1 2

.br 4,475 4,379 52 44

.bs 11 11 1 1

.bw 3 3 1 1

.by 75 72 2 2

.bz 6 6 1 1

.ca 4,942 4,831 65 46

.cat 13 12 1 1

.cc 1 1 1 1

.cd 5 5 1 1

.cf 1 1

.cg 1 1

.ch 1,080 1,052 19 9

.ci 14 13 1 1

.ck 1 1

.cl 916 895 12 9

.cm 19 19 1 1

.cn 4 3 1 1

.co 826 810 8 8

.com 45,594 43,813 1,248 534

.cr 99 96 1 1

.cu 11 11 1 1

.cv 1 1

.cx 1 1

.cz 463 445 15 4

.de 11,208 10,987 147 74

.dj 2 2 1 1

.dk 513 500 9 4

.dm 3 3 1

.do 218 213 2 3

.dz 8 7 1 1

.ec 320 312 3 5

.edu 1 1 1 1

.ee 133 130 2 1

.eg 338 311 8 20

.es 3,974 3,865 61 49

.et 20 19 1 1

.eu 1 1 1

.fi 931 914 13 4

.fj 6 6 1 1

.fm 1 1 1

.fr 7,248 7,086 84 78

.ga 2 2 1 1

.ge 49 44 4 1

.gg 1 1 1

.gh 26 26 1 1

.gi 2 2

.gl 1 1 1

.gm 1 1 1

.gp 4 4 1 1

.gr 528 510 11 6

.gt 59 57 1 1

.gy 4 4 1

.hk 716 644 25 47

.hn 32 31 1 1

.hr 380 365 10 5

.ht 4 4 1

.hu 748 718 23 8

.id 907 856 42 9

.ie 479 471 5 3

.il 671 647 10 14

.im 1 1

.in 3,451 3,257 70 123

.ir 1 1 1

.is 37 37 1 1

.it 5,599 5,463 86 50

.je 1 1

.jm 42 41 1 1

.jo 82 79 2 2

.jp 7,350 6,702 51 598

.ke 53 51 1 1

.kg 9 8 1 1

.kh 8 7 1 1

.ki 1 1

.kr 145 118 16 12

.kw 52 50 1 1

.kz 88 83 3 2

.la 2 2 1 1

.lb 41 40 1 1

.li 2 2

.lk 81 76 3 2

.ls 1 1

.lt 262 252 8 3

.lu 37 37 1 1

.lv 144 140 3 1

.ly 24 22 1 2

.ma 303 282 4 17

.md 70 69 1 1

.me 2 2 1 1

.mg 7 6 1 1

.mk 4 3 1 1

.ml 2 2 1 1

.mn 16 15 1 1

.ms 1 1 1

.mt 35 34 1 1

.mu 21 21 1 1

.mv 11 11 1 1

.mw 2 2 1

.mx 3,485 3,402 41 42

.my 670 640 25 5

.mz 5 5 1 1

.na 3 3 1 1

.ne 1 1 1

.net 27 4 23 1

.ng 57 55 1 2

.ni 14 13 1 1

.nl 1,952 1,897 40 15

.no 528 516 10 2

.np 21 20 1 1

.nr 1 1

.nu 1 1 1

.nz 375 352 5 18

.om 26 25 1 1

.org 36 3 31 1

.pa 10 10 1 1

.pe 735 695 8 32

.ph 1,075 1,049 17 9

.pk 350 328 9 13

.pl 4,866 4,742 89 36

.pn 1 1

.pr 132 129 2 1

.pro 1 1

.ps 42 39 1 2

.pt 719 701 13 5

.py 48 47 1 1

.qa 45 43 1 1

.ro 663 625 20 17

.rs 70 65 3 2

.ru 2,301 2,146 118 36

.rw 3 3 1 1

.sa 265 251 5 9

.sb 1 1

.sc 1 1 1 1

.se 1,249 1,222 20 6

.sg 530 517 9 5

.sh 1 1

.si 199 193 5 2

.sk 289 275 11 4

.sl 1 1 1

.sm 1 1

.sn 15 14 1 1

.st 1 1

.su 1 1

.sv 63 61 1 1

.td 1 1

.tg 1 1 1

.th 904 794 82 28

.tj 1 1 1 1

.tk 1 1 1 1

.tl 1 1 1 1

.tm 1 1 1 1

.to 1 1 1

.tr 1,802 1,712 46 43

.tt 47 46 1 1

.tv 1 1 1

.tw 402 383 7 12

.tz 16 15 1 1

.ua 1,002 961 23 18

.ug 14 13 1 1

.uk 8,283 8,118 108 57

.us 2 1 1

.uy 95 92 2 1

.uz 7 6 1 1

.vc 4 3 1 1

.ve 661 645 7 9

.vg 1 1 1 1

.vi 3 3 1

.vn 628 550 44 35

.vu 1 1

.ws 1 1 1 1

.za 195 189 4 2

.zm 4 3 1 1

.zw 6 5 1 1

undefined 20,476 17,088 635 2,753

Total 168,223 159,398 3,712 5,103

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 954 - - Y
Earth 90 - - Y
FeedFetcher 323 Y - Y
GoogleBot 8,920 Y - Y
GoogleBot? 4,825 - - Y
GoogleBot? 26 - Y Y
Image search 93 - Y -
Mail 9 - Y -
Maps 108 - Y -
Other 226 - - Y
Other 9 - Y -
Other 93 - Y Y
Other 276 Y - -
Other 1,014 Y - Y
Other 3 Y Y Y
Toolbar 160 - - Y
Toolbar 46 - Y -
Toolbar 27 - Y Y
Translate 25 - Y -
Translate 725 Y - Y
Translate 188 Y Y Y
Web search 146,421 - Y -
Web search 16 Y Y -
Wireless 4 - - Y
Wireless 3,638 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Tue, Jan 11, 2011 6:34
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.