Wikimedia Traffic Analysis Report

Wikimedia Traffic Analysis Report - Google requests

Daily averages, based on sample period: 1 Aug 2010 - 31 Aug 2010

This analysis is based on a 1:1000 sampled server log (squids) ⇒ all counts x 1000.
See also: Requests by destination or by origin / Methods / Scripts / Skins / Crawlers / Op.Sys. / Browsers / Google

This report shows all requests to Wikimedia servers where a Google server of service was involved in any way, be it the GoogleBot crawler or FeedFetcher collector scripts that run on Google servers, or a user that follows a link from a Google Web or Google Desktop search results page, or from Google Maps or Google Earth etcetera.
Technically speaking three fields in the squid log records are checked for this: client ip address, referer header and user agent header. A request can originate from an ip address which has been registered by Google and/or it can carry a referer tag that tells us a user clicked a link on a Google results page and/or it can carry an agent string that mentions a Google application which can reasonably be assumed to be genuinely Google's. See bottom of page for further details.
In total Google was somehow involved in 41.0% of daily external page^* requests
Google referred to our sites, through its services including search, maps, and Google Earth, 128,007,000 page views per day, representing 36.4% of our external page requests.
Including all of its different search crawlers and services hosted on its servers, Google itself requested another 16,162,000 page pages per day, representing 4.6% of our external page requests.
* = mime type text/html only

In order of request volume

Requests originating from a Google ip address

Service Total Pages Images Other

GoogleBot 12,688 11,103 588 997

Wireless 3,995 3,946 47 3

Translate 783 783 1 1

Other 455 226 28 202

FeedFetcher 330 92 17 222

Web search 13 12 1 1

Image search 1 1

Toolbar 1 1

Desktop 1 1

KeywordTool 1 1

Total 18,264 16,162 680 1,425

Requests originating from elsewhere

Service Total Pages Images Other

Web search 130,407 124,768 2,551 3,088

GoogleBot? 3,153 2,876 20 257

Desktop 1,052 55 997

Toolbar 297 116 48 133

Other 222 59 38 125

Earth 188 35 153 1

Image search 101 50 50 2

Maps 79 36 42 2

Translate 25 4 15 5

Mail 15 5 9 1

Wireless 6 3 2 1

FeedFetcher 1 1 1 1

Total 135,545 128,007 2,928 4,611

Top level domains

.com 37,718 35,990 859 869

.de 10,235 9,972 168 95

.jp 7,686 6,915 85 686

.uk 7,134 6,958 104 73

.br 5,714 5,561 65 89

.fr 5,220 5,075 63 82

.mx 4,447 4,322 45 80

.it 4,358 4,237 62 59

.ca 3,980 3,866 58 56

.pl 3,274 3,194 52 28

.in 3,163 2,929 68 166

.es 3,085 2,973 49 64

.au 3,022 2,948 51 23

.co 2,379 2,314 22 43

.ar 1,749 1,682 25 41

.nl 1,602 1,552 32 18

.ph 1,354 1,290 43 20

.ru 1,248 1,145 87 15

.cl 1,205 1,164 20 21

.pe 1,090 1,011 14 65

.th 1,089 890 150 50

.tr 1,053 992 24 38

.se 1,017 995 15 7

.be 976 945 17 15

.ch 969 941 17 11

.id 933 787 135 11

.at 824 799 15 10

.fi 820 807 7 5

.my 732 653 69 10

.vn 612 503 63 46

.hu 566 539 17 9

.ve 566 547 7 12

.hk 554 485 28 41

.pt 509 496 8 5

.ro 508 479 14 15

.il 506 484 8 14

.nz 505 482 8 15

.sg 503 476 22 5

.ua 488 476 6 7

.dk 468 454 10 4

.ie 407 398 4 5

.gr 381 369 7 4

.no 373 364 7 2

.za 343 329 8 7

.bg 340 325 7 9

.cz 306 292 11 3

.ec 280 270 3 7

.tw 268 251 5 12

.pk 262 244 7 12

.hr 260 252 6 2

.gt 251 242 3 7

.do 243 234 2 6

.eg 196 179 5 13

.sk 190 178 8 4

.cr 176 171 3 3

.uy 175 169 2 4

.bo 168 156 3 9

.sa 166 154 4 8

.lt 161 156 4 1

.sv 158 154 1 3

.si 154 149 3 2

.pr 146 143 2 1

.kr 145 102 23 21

.ma 143 132 2 9

.ae 140 137 2 1

.py 97 93 1 2

.ee 93 91 1 1

.lv 90 87 2 1

.ba 84 81 2 1

.lk 72 68 2 2

.hn 68 66 1 1

.ng 62 59 1 3

.ni 53 51 1 1

.ke 52 51 1 1

.by 50 49 1 1

.jo 43 40 1 1

.rs 43 40 1 1

.kz 41 40 1 1

.kw 40 38 1 1

.tt 35 34 1 1

.qa 35 33 1 1

.lu 34 33 1 1

.md 31 30 1 1

.is 28 28 1 1

.jm 28 28 1 1

.mt 28 27 1 1

.lb 27 26 1 1

.mu 26 25 1 1

.bh 25 24 1 1

.np 25 23 1 1

.pa 23 22 1 1

.gh 23 22 1 1

.ge 22 20 1 1

.az 18 17 1 1

.ps 17 15 1 1

.om 17 16 1 1

.cm 14 13 1 1

.org 14 2 12 1

.ug 13 12 1 1

.ly 13 12 1 1

.ci 12 12 1 1

.am 12 11 1 1

.mz 12 11 1 1

.et 12 11 1 1

.mv 11 10 1 1

.cu 11 11 1 1

.bn 11 11 1 1

.sn 10 10 1 1

.tz 10 9 1 1

.bs 10 10 1 1

.fj 8 8 1 1

.mn 8 8 1 1

.cat 8 7 1 1

.net 8 1 6 1

.zw 7 6 1 1

.na 6 5 1 1

.bw 6 6 1 1

.kh 6 6 1

.rw 5 5 1 1

.mg 5 5 1 1

.zm 5 4 1 1

.bd 4 4 1 1

.dz 4 4 1 1

.bz 4 4 1

.dm 3 3 1

.af 3 2 1 1

.kg 3 3 1

.cd 3 3 1

.cn 3 3 1 1

.uz 3 3 1 1

.ht 3 3 1 1

.gy 3 3 1

.vc 3 3 1

.gp 3 3 1

.bf 2 2

.gi 2 2

.li 2 2 1

.ag 2 2 1

.vi 2 2

.ml 2 2 1 1

.dj 2 1 1 1

.ne 1 1

.biz 1 1 1 1

.gl 1 1

.ls 1 1 1

.la 1 1 1

.ga 1 1 1

.sb 1 1 1 1

.ai 1 1 1

.me 1 1 1 1

.vg 1 1

.as 1 1 1

.sc 1 1

.vu 1 1

.bj 1 1 1 1

.gm 1 1 1

.to 1 1 1

.eu 1 1 1 1

.tl 1 1

.mk 1 1

.fm 1 1 1

.ws 1 1 1

.ad 1 1

.tg 1 1 1

.sm 1 1

.gg 1 1 1

.ki 1 1

.nu 1 1 1

.st 1 1 1

.ck 1 1

.ao 1 1

.tm 1 1 1

.ms 1 1

.pn 1 1

.ir 1 1

.td 1 1

.cg 1 1 1

.sl 1 1

.us 1 1 1

.tv 1 1 1

.bi 1 1 1

.tk 1 1 1

.nr 1 1

.edu 1 1

.mw 1 1 1

.ac 1 1

.je 1 1

.cc 1 1 1

.cf 1 1

.im 1 1

undefined 22,464 18,730 814 2,920

Total 153,804 144,166 3,602 6,027

In alphabetical order

Requests originating from a Google ip address

Service Total Pages Images Other

Wireless 3,995 3,946 47 3

Web search 13 12 1 1

Translate 783 783 1 1

Toolbar 1 1

Other 455 226 28 202

KeywordTool 1 1

Image search 1 1

GoogleBot 12,688 11,103 588 997

FeedFetcher 330 92 17 222

Desktop 1 1

Total 18,264 16,162 680 1,425

Requests originating from elsewhere

Service Total Pages Images Other

Wireless 6 3 2 1

Web search 130,407 124,768 2,551 3,088

Translate 25 4 15 5

Toolbar 297 116 48 133

Other 222 59 38 125

Maps 79 36 42 2

Mail 15 5 9 1

Image search 101 50 50 2

GoogleBot? 3,153 2,876 20 257

FeedFetcher 1 1 1 1

Earth 188 35 153 1

Desktop 1,052 55 997

Total 135,545 128,007 2,928 4,611

Top level domains

.ac 1 1

.ad 1 1

.ae 140 137 2 1

.af 3 2 1 1

.ag 2 2 1

.ai 1 1 1

.am 12 11 1 1

.ao 1 1

.ar 1,749 1,682 25 41

.as 1 1 1

.at 824 799 15 10

.au 3,022 2,948 51 23

.az 18 17 1 1

.ba 84 81 2 1

.bd 4 4 1 1

.be 976 945 17 15

.bf 2 2

.bg 340 325 7 9

.bh 25 24 1 1

.bi 1 1 1

.biz 1 1 1 1

.bj 1 1 1 1

.bn 11 11 1 1

.bo 168 156 3 9

.br 5,714 5,561 65 89

.bs 10 10 1 1

.bw 6 6 1 1

.by 50 49 1 1

.bz 4 4 1

.ca 3,980 3,866 58 56

.cat 8 7 1 1

.cc 1 1 1

.cd 3 3 1

.cf 1 1

.cg 1 1 1

.ch 969 941 17 11

.ci 12 12 1 1

.ck 1 1

.cl 1,205 1,164 20 21

.cm 14 13 1 1

.cn 3 3 1 1

.co 2,379 2,314 22 43

.com 37,718 35,990 859 869

.cr 176 171 3 3

.cu 11 11 1 1

.cz 306 292 11 3

.de 10,235 9,972 168 95

.dj 2 1 1 1

.dk 468 454 10 4

.dm 3 3 1

.do 243 234 2 6

.dz 4 4 1 1

.ec 280 270 3 7

.edu 1 1

.ee 93 91 1 1

.eg 196 179 5 13

.es 3,085 2,973 49 64

.et 12 11 1 1

.eu 1 1 1 1

.fi 820 807 7 5

.fj 8 8 1 1

.fm 1 1 1

.fr 5,220 5,075 63 82

.ga 1 1 1

.ge 22 20 1 1

.gg 1 1 1

.gh 23 22 1 1

.gi 2 2

.gl 1 1

.gm 1 1 1

.gp 3 3 1

.gr 381 369 7 4

.gt 251 242 3 7

.gy 3 3 1

.hk 554 485 28 41

.hn 68 66 1 1

.hr 260 252 6 2

.ht 3 3 1 1

.hu 566 539 17 9

.id 933 787 135 11

.ie 407 398 4 5

.il 506 484 8 14

.im 1 1

.in 3,163 2,929 68 166

.ir 1 1

.is 28 28 1 1

.it 4,358 4,237 62 59

.je 1 1

.jm 28 28 1 1

.jo 43 40 1 1

.jp 7,686 6,915 85 686

.ke 52 51 1 1

.kg 3 3 1

.kh 6 6 1

.ki 1 1

.kr 145 102 23 21

.kw 40 38 1 1

.kz 41 40 1 1

.la 1 1 1

.lb 27 26 1 1

.li 2 2 1

.lk 72 68 2 2

.ls 1 1 1

.lt 161 156 4 1

.lu 34 33 1 1

.lv 90 87 2 1

.ly 13 12 1 1

.ma 143 132 2 9

.md 31 30 1 1

.me 1 1 1 1

.mg 5 5 1 1

.mk 1 1

.ml 2 2 1 1

.mn 8 8 1 1

.ms 1 1

.mt 28 27 1 1

.mu 26 25 1 1

.mv 11 10 1 1

.mw 1 1 1

.mx 4,447 4,322 45 80

.my 732 653 69 10

.mz 12 11 1 1

.na 6 5 1 1

.ne 1 1

.net 8 1 6 1

.ng 62 59 1 3

.ni 53 51 1 1

.nl 1,602 1,552 32 18

.no 373 364 7 2

.np 25 23 1 1

.nr 1 1

.nu 1 1 1

.nz 505 482 8 15

.om 17 16 1 1

.org 14 2 12 1

.pa 23 22 1 1

.pe 1,090 1,011 14 65

.ph 1,354 1,290 43 20

.pk 262 244 7 12

.pl 3,274 3,194 52 28

.pn 1 1

.pr 146 143 2 1

.ps 17 15 1 1

.pt 509 496 8 5

.py 97 93 1 2

.qa 35 33 1 1

.ro 508 479 14 15

.rs 43 40 1 1

.ru 1,248 1,145 87 15

.rw 5 5 1 1

.sa 166 154 4 8

.sb 1 1 1 1

.sc 1 1

.se 1,017 995 15 7

.sg 503 476 22 5

.si 154 149 3 2

.sk 190 178 8 4

.sl 1 1

.sm 1 1

.sn 10 10 1 1

.st 1 1 1

.sv 158 154 1 3

.td 1 1

.tg 1 1 1

.th 1,089 890 150 50

.tk 1 1 1

.tl 1 1

.tm 1 1 1

.to 1 1 1

.tr 1,053 992 24 38

.tt 35 34 1 1

.tv 1 1 1

.tw 268 251 5 12

.tz 10 9 1 1

.ua 488 476 6 7

.ug 13 12 1 1

.uk 7,134 6,958 104 73

.us 1 1 1

.uy 175 169 2 4

.uz 3 3 1 1

.vc 3 3 1

.ve 566 547 7 12

.vg 1 1

.vi 2 2

.vn 612 503 63 46

.vu 1 1

.ws 1 1 1

.za 343 329 8 7

.zm 5 4 1 1

.zw 7 6 1 1

undefined 22,464 18,730 814 2,920

Total 153,804 144,166 3,602 6,027

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges
Agents: as for genuine agent strings: too many crawlers indentify themselves as 'GoogleBot' to take this at face value. They are accepted as genuine Google crawler requests only when the ip address matches a known range (see above). Other records that mention GoogleBot are counted as GoogleBot? (question mark, as this may include partners, like DoCoMo). However when the agent string mentions Google Desktop or Google Earth this is always accepted
Service: the service name is based on the agent string (plus for GoogleBot check for ip address, see above), if this is inconclusive it is based on the referer string.
Here is detailed breakdown per service of indicators that pointed to Google (total ≥ 3)

Service Total Originating from
Google ip address Referer mentions
Google url Agent mentions
Google service

Desktop 1,052 - - Y
Earth 188 - - Y
FeedFetcher 330 Y - Y
GoogleBot 12,688 Y - Y
GoogleBot? 3,146 - - Y
GoogleBot? 7 - Y Y
Image search 101 - Y -
Mail 15 - Y -
Maps 79 - Y -
Other 184 - - Y
Other 7 - Y -
Other 31 - Y Y
Other 258 Y - -
Other 197 Y - Y
Toolbar 207 - - Y
Toolbar 59 - Y -
Toolbar 30 - Y Y
Translate 24 - Y -
Translate 590 Y - Y
Translate 194 Y Y Y
Web search 130,407 - Y -
Web search 13 Y Y -
Wireless 6 - - Y
Wireless 3,995 Y - Y

Top Level Domain 'undefined': requests with top level domain 'undefined' are nearly all requests from anonymous ip addresses (crawler and other services)
Note: averages below 1 are always rounded up to 1

Generated on Tue, Oct 19, 2010 3:22
Author:Erik Zachte (Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.