Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Jan 2011 - 31 Jan 2011

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 43.9% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 163,289,000 page views per day, representing 39.8% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 16,785,000 page pages per day, representing 4.1% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 12,409 11,007 554 848

Wireless 4,036 3,968 63 5

Other 1,573 715 279 579

Translate 993 993 1 1

FeedFetcher 316 88 17 211

Web search 15 14 1 1

Image search 1 1

Toolbar 1 1

Desktop 1 1

Total 19,342 16,785 913 1,644

Requests originating from elsewhere

Service Total Pages Images Other

Web search 164,693 159,538 2,691 2,464

GoogleBot? 3,698 3,391 35 272

Desktop 962 33 1 928

Other 376 127 133 116

Toolbar 245 93 29 123

Maps 119 28 90 1

Earth 94 29 65 1

Image search 80 39 40 1

Translate 28 4 18 5

Mail 9 4 5 1

Wireless 3 2 1 1

FeedFetcher 1 1 1

Total 170,308 163,289 3,107 3,910

Top level domains

.com 51,637 49,727 1,338 572

.de 13,534 13,281 157 96

.uk 10,272 10,065 136 71

.fr 8,421 8,244 90 87

.jp 8,190 7,470 52 668

.it 6,767 6,604 103 60

.pl 5,807 5,661 104 42

.ca 5,550 5,423 76 51

.es 4,684 4,559 70 55

.br 4,159 4,068 51 40

.in 3,593 3,388 76 129

.mx 3,337 3,258 34 45

.au 2,478 2,435 28 14

.ru 2,397 2,254 110 33

.nl 2,299 2,230 49 19

.tr 1,660 1,590 35 34

.ph 1,535 1,494 25 15

.se 1,454 1,423 23 7

.be 1,400 1,362 26 12

.ch 1,287 1,255 21 11

.at 1,257 1,226 21 10

.ar 1,061 1,030 17 14

.id 1,045 989 43 12

.fi 1,037 1,018 14 4

.ua 1,011 973 21 18

.th 997 884 82 31

.hu 918 883 26 9

.pt 915 893 15 8

.co 911 895 9 8

.ro 897 851 24 22

.my 741 708 26 7

.ve 700 685 5 10

.il 693 671 10 12

.cl 676 661 8 7

.no 650 634 13 4

.hk 636 590 14 31

.dk 630 614 12 4

.cz 628 604 20 4

.gr 618 598 13 7

.sg 587 572 10 5

.vn 558 486 42 30

.ie 557 547 6 4

.bg 551 528 12 11

.hr 434 420 11 4

.pe 410 391 3 15

.nz 409 394 5 10

.tw 383 365 7 11

.sk 373 353 14 6

.pk 365 343 8 14

.lt 318 306 8 5

.za 283 272 7 4

.ma 258 241 3 14

.si 237 228 6 3

.sa 234 222 5 8

.eg 217 201 5 11

.ec 213 207 2 3

.ae 203 198 4 1

.ee 156 152 3 1

.lv 151 145 4 1

.do 150 146 1 2

.kr 145 120 15 10

.ba 136 130 4 2

.pr 124 121 2 1

.gt 113 111 1 2

.by 107 102 3 2

.kz 87 80 4 2

.lk 87 82 3 2

.cr 77 75 1 1

.rs 72 68 2 1

.md 71 69 1 1

.ng 65 62 1 2

.uy 63 61 1 1

.sv 62 60 1 1

.ke 61 59 1 1

.jo 59 56 2 1

.jm 57 55 1 1

.tt 52 51 1 1

.qa 50 47 1 2

.lu 49 47 1 1

.org 47 2 43 2

.kw 46 45 1 1

.ge 45 41 3 1

.mt 45 44 1 1

.is 44 42 1 1

.lb 44 42 1 1

.bo 40 38 1 1

.py 36 35 1 1

.az 36 32 2 2

.bh 31 30 1 1

.mu 30 29 1 1

.hn 29 28 1 1

.gh 28 27 1 1

.net 27 4 22 1

.ps 26 24 1 1

.am 26 24 1 1

.om 25 24 1 1

.cm 23 21 1 1

.np 22 21 1 1

.ly 20 18 1 1

.tz 18 17 1 1

.sn 17 16 1 1

.ni 17 17 1 1

.ug 15 15 1 1

.ci 15 15 1 1

.mn 14 13 1 1

.bs 14 14 1 1

.cat 14 13 1 1

.et 14 13 1 1

.bn 14 14 1 1

.pa 11 11 1 1

.dz 11 10 1 1

.mv 10 10 1 1

.mg 9 9 1 1

.cu 9 9 1 1

.bz 8 8 1 1

.uz 8 7 1 1

.gy 8 7 1 1

.fj 7 7 1 1

.bw 7 7 1 1

.kh 7 7 1 1

.zw 7 6 1 1

.kg 6 6 1 1

.mz 6 6 1 1

.na 5 5 1 1

.cd 5 5 1 1

.ht 5 4 1 1

.bd 4 3 1 1

.rw 4 4 1 1

.dm 4 4 1

.af 4 3 1 1

.cn 4 4 1 1

.ag 4 4 1

.vi 4 4 1 1

.vc 4 4 1

.gp 4 4 1 1

.bf 3 3 1 1

.gi 3 3 1

.li 3 3 1

.ml 3 3 1

.mw 3 2 1 1

.zm 3 3 1

.mk 3 2 1 1

.biz 2 1 1 1

.ao 2 2 1

.la 2 2 1 1

.ga 2 2 1

.me 2 2 1 1

.us 2 1 1

.bj 2 1 1 1

.gm 2 1 1

.eu 2 2

.tg 2 2 1 1

.dj 2 2 1

.ne 1 1 1

.gg 1 1 1

.gl 1 1 1

.ls 1 1 1

.tm 1 1 1

.ai 1 1

.vg 1 1 1 1

.ir 1 1 1

.cg 1 1

.bi 1 1

.as 1 1

.tk 1 1 1

.sc 1 1

.to 1 1

.cc 1 1 1

.tl 1 1 1

.fm 1 1 1

.ws 1 1 1

.ad 1 1 1

.sm 1 1 1 1

.ki 1 1 1

.nu 1 1 1

.tel 1 1

.st 1 1

.hm 1 1

.ck 1 1 1

.tc 1 1

.ms 1 1 1

.pn 1 1

.sb 1 1 1

.td 1 1 1

.sl 1 1 1

.tv 1 1

.nr 1 1

.edu 1 1 1

.ac 1 1

.je 1 1

.sh 1 1

.cf 1 1

.tj 1 1 1

.im 1 1 1

undefined 23,540 19,773 719 3,048

Total 189,656 180,066 4,014 5,547

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 4,036 3,968 63 5

Web search 15 14 1 1

Translate 993 993 1 1

Toolbar 1 1

Other 1,573 715 279 579

Image search 1 1

GoogleBot 12,409 11,007 554 848

FeedFetcher 316 88 17 211

Desktop 1 1

Total 19,342 16,785 913 1,644

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 3 2 1 1

Web search 164,693 159,538 2,691 2,464

Translate 28 4 18 5

Toolbar 245 93 29 123

Other 376 127 133 116

Maps 119 28 90 1

Mail 9 4 5 1

Image search 80 39 40 1

GoogleBot? 3,698 3,391 35 272

FeedFetcher 1 1 1

Earth 94 29 65 1

Desktop 962 33 1 928

Total 170,308 163,289 3,107 3,910

Top level domains

.ac 1 1

.ad 1 1 1

.ae 203 198 4 1

.af 4 3 1 1

.ag 4 4 1

.ai 1 1

.am 26 24 1 1

.ao 2 2 1

.ar 1,061 1,030 17 14

.as 1 1

.at 1,257 1,226 21 10

.au 2,478 2,435 28 14

.az 36 32 2 2

.ba 136 130 4 2

.bd 4 3 1 1

.be 1,400 1,362 26 12

.bf 3 3 1 1

.bg 551 528 12 11

.bh 31 30 1 1

.bi 1 1

.biz 2 1 1 1

.bj 2 1 1 1

.bn 14 14 1 1

.bo 40 38 1 1

.br 4,159 4,068 51 40

.bs 14 14 1 1

.bw 7 7 1 1

.by 107 102 3 2

.bz 8 8 1 1

.ca 5,550 5,423 76 51

.cat 14 13 1 1

.cc 1 1 1

.cd 5 5 1 1

.cf 1 1

.cg 1 1

.ch 1,287 1,255 21 11

.ci 15 15 1 1

.ck 1 1 1

.cl 676 661 8 7

.cm 23 21 1 1

.cn 4 4 1 1

.co 911 895 9 8

.com 51,637 49,727 1,338 572

.cr 77 75 1 1

.cu 9 9 1 1

.cz 628 604 20 4

.de 13,534 13,281 157 96

.dj 2 2 1

.dk 630 614 12 4

.dm 4 4 1

.do 150 146 1 2

.dz 11 10 1 1

.ec 213 207 2 3

.edu 1 1 1

.ee 156 152 3 1

.eg 217 201 5 11

.es 4,684 4,559 70 55

.et 14 13 1 1

.eu 2 2

.fi 1,037 1,018 14 4

.fj 7 7 1 1

.fm 1 1 1

.fr 8,421 8,244 90 87

.ga 2 2 1

.ge 45 41 3 1

.gg 1 1 1

.gh 28 27 1 1

.gi 3 3 1

.gl 1 1 1

.gm 2 1 1

.gp 4 4 1 1

.gr 618 598 13 7

.gt 113 111 1 2

.gy 8 7 1 1

.hk 636 590 14 31

.hm 1 1

.hn 29 28 1 1

.hr 434 420 11 4

.ht 5 4 1 1

.hu 918 883 26 9

.id 1,045 989 43 12

.ie 557 547 6 4

.il 693 671 10 12

.im 1 1 1

.in 3,593 3,388 76 129

.ir 1 1 1

.is 44 42 1 1

.it 6,767 6,604 103 60

.je 1 1

.jm 57 55 1 1

.jo 59 56 2 1

.jp 8,190 7,470 52 668

.ke 61 59 1 1

.kg 6 6 1 1

.kh 7 7 1 1

.ki 1 1 1

.kr 145 120 15 10

.kw 46 45 1 1

.kz 87 80 4 2

.la 2 2 1 1

.lb 44 42 1 1

.li 3 3 1

.lk 87 82 3 2

.ls 1 1 1

.lt 318 306 8 5

.lu 49 47 1 1

.lv 151 145 4 1

.ly 20 18 1 1

.ma 258 241 3 14

.md 71 69 1 1

.me 2 2 1 1

.mg 9 9 1 1

.mk 3 2 1 1

.ml 3 3 1

.mn 14 13 1 1

.ms 1 1 1

.mt 45 44 1 1

.mu 30 29 1 1

.mv 10 10 1 1

.mw 3 2 1 1

.mx 3,337 3,258 34 45

.my 741 708 26 7

.mz 6 6 1 1

.na 5 5 1 1

.ne 1 1 1

.net 27 4 22 1

.ng 65 62 1 2

.ni 17 17 1 1

.nl 2,299 2,230 49 19

.no 650 634 13 4

.np 22 21 1 1

.nr 1 1

.nu 1 1 1

.nz 409 394 5 10

.om 25 24 1 1

.org 47 2 43 2

.pa 11 11 1 1

.pe 410 391 3 15

.ph 1,535 1,494 25 15

.pk 365 343 8 14

.pl 5,807 5,661 104 42

.pn 1 1

.pr 124 121 2 1

.ps 26 24 1 1

.pt 915 893 15 8

.py 36 35 1 1

.qa 50 47 1 2

.ro 897 851 24 22

.rs 72 68 2 1

.ru 2,397 2,254 110 33

.rw 4 4 1 1

.sa 234 222 5 8

.sb 1 1 1

.sc 1 1

.se 1,454 1,423 23 7

.sg 587 572 10 5

.sh 1 1

.si 237 228 6 3

.sk 373 353 14 6

.sl 1 1 1

.sm 1 1 1 1

.sn 17 16 1 1

.st 1 1

.sv 62 60 1 1

.tc 1 1

.td 1 1 1

.tel 1 1

.tg 2 2 1 1

.th 997 884 82 31

.tj 1 1 1

.tk 1 1 1

.tl 1 1 1

.tm 1 1 1

.to 1 1

.tr 1,660 1,590 35 34

.tt 52 51 1 1

.tv 1 1

.tw 383 365 7 11

.tz 18 17 1 1

.ua 1,011 973 21 18

.ug 15 15 1 1

.uk 10,272 10,065 136 71

.us 2 1 1

.uy 63 61 1 1

.uz 8 7 1 1

.vc 4 4 1

.ve 700 685 5 10

.vg 1 1 1 1

.vi 4 4 1 1

.vn 558 486 42 30

.ws 1 1 1

.za 283 272 7 4

.zm 3 3 1

.zw 7 6 1 1

undefined 23,540 19,773 719 3,048

Total 189,656 180,066 4,014 5,547

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 962 - - Y
Earth 94 - - Y
FeedFetcher 316 Y - Y
GoogleBot 12,409 Y - Y
GoogleBot? 3,686 - - Y
GoogleBot? 12 - Y Y
Image search 80 - Y -
Mail 9 - Y -
Maps 119 - Y -
Other 262 - - Y
Other 11 - Y -
Other 103 - Y Y
Other 164 Y - -
Other 1,405 Y - Y
Other 3 Y Y Y
Toolbar 161 - - Y
Toolbar 53 - Y -
Toolbar 30 - Y Y
Translate 27 - Y -
Translate 791 Y - Y
Translate 203 Y Y Y
Web search 164,693 - Y -
Web search 15 Y Y -
Wireless 3 - - Y
Wireless 4,036 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Tue, Feb 15, 2011 2:17
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.